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Abstract 

Performance-Based  Logistics  (PBL)  is  DoD' s  preferred  product  support  approach 
in  order  to  satisfy  vitally-important  war  fighter  needs.  The  elasticity  of  these  needs  makes 
it  impossible  to  build  a  one-fits-all  template  PBL  contract  with  the  contractors.  Thus, 
both  economic  metrics  such  as  Cost  per  Unit  Usage  and  non-economic  metrics  such  as 
Availability,  Reliability,  Logistics  Foot  Print  and  Logistics  Response  Time  can  be 
defined  and  tailored  according  to  each  individual  program. 

However,  evaluation  of  the  success  of  a  PBL  contract  gets  harder  proportional  to 
the  increased  number  of  different  types  of  metrics.  This  research  suggests  an  integrated 
model  approach  for  combining  multiple  criteria  when  assessing  the  success  of  a  PBL 
contract. 

This  approach  may  be  used  in  operational-level  decisions  such  as  reward  and 
penalty  decisions  within  a  contract,  or  strategic-level  decisions  such  as  extending  the 
contract  with  the  same  contractor,  terminating  the  contract  and  negotiating 
with  other  contractors. 
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A  4-STEP  PROCESS  EVALUATION  MODEL  TO  ASSESS  THE  SUCCESS  OF 


PERFORMANCE  BASED  LOGISTICS  CONTRACTS 


I.  Introduction 


Chapter  Overview 

This  thesis  investigates  and  proposes  an  alternative  approach  to  monitor  the 
Perfonnance  Based  Logistics  (PBL)  supported  systems  both  at  the  individual  metrics 
level  and  at  the  whole  system  level.  For  this  reason,  the  defined  metrics  of  the  PBL 
contracts  are  integrated  in  a  mathematical  model  and  that  model  is  integrated  into  the 
current  evaluation  processes  in  order  to  build  a  theoretical  model  that  would  fill  the 
defined  gaps  of  the  PBL  evaluation  processes.  In  addition  to  that,  opening  a  discussion 
among  the  academics  and  the  acquisition  professionals  is  intended. 

First  of  all,  the  current  study  starts  with  reviewing  the  published  sources  and  the 
experts’  opinions  in  order  to  build  our  foundation  of  PBL.  Then,  the  gaps  of  the  PBL 
literature  are  defined  and  the  focus  of  the  current  study  is  stated.  The  focus  of  this  study 
is  to  investigate  and  propose  a  reasonable  approach  for  the  evaluation  process  in  PBL. 

According  to  the  knowledge  that  has  been  gained  until  that  point,  the  scope  and 
the  assumptions  of  the  study  are  defined  and  the  investigative  questions  are  determined. 
Next  the  methodology  is  described  accordingly  and  the  proposed  model  is  built.  Then  the 


1 


model  is  evaluated,  verified  and  validated.  Next  the  model  is  applied  to  two  real-life 
cases  and  the  applicability  and  the  usefulness  is  assessed. 

Finally,  the  gathered  results  are  analyzed  and  the  core  investigative  questions 
answered  with  the  relevant  findings.  Additionally,  the  experienced  limitations  and 
possible  future  studies  are  addressed. 

This  chapter  includes  the  background,  problem  statement,  research  objectives, 
investigative  questions,  research  focus,  methodology,  assumptions  /  limitations  and 
research  implications. 

Background 

Beyond  any  question  the  most  remarkable  global  developments  of  the  end  of  the 
20th  Century  and  the  beginning  of  the  2 1st  Century  were:  the  ending  of  the  Cold  War 
which  resulted  in  the  dissolution  of  the  Union  of  Soviet  Socialist  Republics  (USSR)  and 
the  Warsaw  Treaty  and  the  9/1 1  attacks,  that  initiated  the  beginning  of  the  “Global  War 
on  Terrorism”  (GWOT).  The  necessities  of  the  new  era  mandated  every  country  to 
allocate  their  resources  relative  to  the  new  war  concepts  that  are  evolved  from  the 
previous  ones.  Under  these  circumstances,  PBL  is  defined  as:  “Department  of  Defense 
(DoD)’s  preferred  product  support  approach.”  (DoD  Directive  5000.1,  2003)  Basically, 
PBL  may  be  defined  as  buying  the  performance  instead  of  buying  the  materials  (Mahon, 
2007).  It  may  seem  an  easy  explanation  but  has  created  a  lot  of  confusion  in  the  minds  of 
many  acquisition  professionals  from  both  the  Government’s  side  and  the  contractors’ 
side.  First  of  all,  PBL  was  born  under  the  highly  elastic  conditions  described  above, 
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which  means  it  must  be  highly  elastic  also.  That’s  why  the  vision  of  PBL  can  be 
translated  as  answering  intensively-elastic  needs  of  the  war  fighter  instantly  with  a 
reasonable  cost  for  the  nation.  So  there’s  no  magic  template  for  implementing  a  PBL 
program  and  every  program  is  unique  according  to  the  defined  needs  of  the  war  fighter.  If 
there’s  no  template,  how  should  it  be  implemented  correctly?  How  can  anybody  be  sure 
about  earning  the  best  results  via  using  PBL?  At  what  extent  can  we  trust  a  contractor  and 
give  him  authority  over  our  weapon  system?  Although  the  Government  has  implemented 
a  series  of  publications  to  clarify  the  confusions  mentioned  above  since  the  first 
introduction  of  the  PBL,  there’s  still  some  gaps  that  are  creating  difficulties  for  the 
stakeholders  of  the  PBL  programs. 

Besides  the  Government  officials,  the  academics  also  contributed  to  the  efforts  of 
defining  the  best  way  to  implement  PBL  programs,  meanwhile  putting  together  the 
lessons  learned  from  the  previous  experiences  and  offering  some  general  solutions  to  the 
problems  that  occurred  then.  However,  it  is  very  constrained  for  the  academics  to  do  their 
research  comprehensively  because  of  some  difficulties  such  as:  reaching  the  real  data  of 
the  PBL  programs,  differences  of  the  military  system  from  the  commercial  system, 
diversity  of  the  PBL  programs  and  the  privacy  necessities  of  the  contractors  in  order  to 
keep  their  competitive  advantages  in  the  market.  According  to  a  study,  seven  key  barriers 
and  seven  enablers  are  defined  as  depicted  in  Table  1  (Devries,  2004). 

While  there’s  an  increased  effort  to  implement  PBL  for  the  Government’s  product 
support,  it  hasn’t  been  achieved  yet.  According  to  the  latest  report  of  the  Government 
Accountability  Office  (US  GAO)  depending  on  a  research  that  compares  the  decision¬ 
making  via  using  performance  measures  between  1997  and  2007 
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However,  despite  having  more  perfonnance  measures  available,  federal 
managers’  reported  use  of  perfonnance  infonnation  in  management  decision¬ 
making  has  not  changed  significantly.  (US  GAO  July  2008,  Highlights) 


Table  1.  The  Barriers  and  the  Enablers  of  PBL  Implementation 


(Adapted  from  Devries,  2004) 


BARRIERS  OF  PBL 
IMPLEMENTATION 

ENABLERS  OF  PBL 
IMPLEMENTATION 

Funding  restrictions  /inflexibility 
(e.g.,  Working  Capital  Fund,  various 
appropriations 

/transfer  and  expiring  funds  rules,  limited 
Program  Manager  [PM] 

Supply  Chain  Management 

(e.g.,  end-to-end  customer  support,  enterprise 

integration). 

Statutory  /regulatory  requirements 
(e.g..  Title  10,  service  policies). 

Strategic  alliances  /partnerships 
(e.g.,  depot  partnering,  joint  ventures) 

Old  paradigms  /culture 

(e.g.,  organic  versus  commercial,  parts 

management  versus 

performance  management,  minimize 

contractors  on  the  battlefield). 

Performance-  based  contracting 
(e.g.,  incentivizing  performance) 

Existing  infrastructure  /bureaucracy  (e.g.,  PM 
office  structure,  stovepiping,  short  PM  tours). 

Performance-based  metrics 

Technical  data  Rights  issues. 

Total  Life  Cycle  Systems  Management 
(TLCSM)  perspective 

Lack  of  PBL  awareness  /training 

Adoption  of  Commercial  Off-the-Shelf 
(COTS) 

/Best  Commercial  Practices 

Inability  to  incentivize  organic  providers. 

Reduction  in  Total  Ownership  Cost  (RTOC) 
initiative 

Problem  Statement 


According  to  the  research  findings  of  Devries  (2004)  performance  metrics  play 
the  most  important  role  in  the  process  of  PBL  implementation  (See  Figure  1). 
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Figure  1.  The  Enablers  of  PBL  implementation 

(Gathered  from  Devries,  2004) 


This  research,  which  depends  on  both  the  published  literature  and  face-  to-face 
meetings  with  the  experts,  also  supported  that  there  are  many  issues  from  defining  the 
metrics  to  evaluation  of  them  in  the  PBL  implementation  process. 

By  definition,  PBL  aims  at  reaching  the  desired  outcomes  via  the  capabilities  of 
contractors  without  mandating  how  to  achieve  those  goals.  Basically,  the  desired 
outcomes  and  the  relevant  metrics  should  be  defined  by  the  customer  for  the  first  step. 
Then,  an  agreement  with  the  contractor  should  be  achieved  with  the  contractor. 

Next,  the  data  of  the  defined  metrics  should  be  collected.  Finally,  the  collected  data  of 
assessed  values  of  the  metrics  should  be  evaluated.  Otherwise,  the  performance  of  the 
system  may  not  be  monitored  and  managed  in  a  healthy  manner. 

Therefore,  an  acceptable,  general  and  easy-to-understand  approach  which  may  be 
used  to  evaluate  the  defined  metrics  in  order  to  monitor  the  system  is  investigated  in  this 
study. 
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Research  Objectives 


First  of  all,  the  basic  aim  of  any  system  is  achieving  some  targeted  goals.  For  this 
reason,  the  metrics  should  be  defined  according  to  the  targeted  goals,  data  must  be 
collected  and  a  systematic  and  reasonable  approach  should  be  chosen  in  order  to  evaluate 
the  system.  Without  a  doubt,  PBL-supported  systems  are  not  exempted  from  these 
fundamental  processes.  There  may  not  be  a  template  to  implement  the  PBL  contracts,  so 
a  common  systematic  approach  should  be  built. 

In  conclusion,  in  this  study  a  reasonable  model  for  evaluation  of  the  defined 
metrics  to  monitor  the  system  is  proposed  to  make  healthy  decisions.  As  a  result,  the 
following  investigative  questions  have  to  be  answered. 

Investigative  Questions 

1)  What  are  the  general  metrics  defined  in  the  contracts? 

Actually,  there’s  neither  a  standard  template  for  PBL  contracts,  nor  standardized 
metrics.  However,  relevant  to  their  knowledge  about  PBL,  the  Defense  Acquisition 
University  (DAU)  mentioned  some  useful  metrics  as:  Operational  Availability  (Ao), 
Operational  Reliability  (R),  Cost  per  Unit  Usage  (CUU),  Logistics  Footprint  (LF)  and 
Logistics  Response  Time  (LRT)  in  their  published  book  of  “Performance  Based 
Logistics:  A  Program  Manager's  Product  Support  Guide”,  also  known  as  PBL  Guide. 
(DAU,  2005) 
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However,  some  academics  mentioned  that  these  metrics  are  high  level  composite 
metrics  (Kumar  et  ah,  2007).  So  in  each  PBL  contract,  some  or  all  of  these  metrics  should 
be  tailored  according  to  the  desired  goals  and  would  be  negotiated  with  the  contractor/s. 

2)  What  is  the  general  current  evaluation  method  of  the  metrics? 

According  to  the  experts’  opinions,  in  most  of  the  PBL  contracts,  the  metrics  are 
evaluated  by  a  two  step  process  that  is: 

a)  Collecting  assessed  values  of  the  metrics  within  the  contractually-defined  terms, 
(ex:  average  monthly  values) 

b)  Comparing  assessed  values  of  the  metrics  with  the  contractually  targeted  numbers 
of  each  metric  individually. 1 

3)  What  are  the  gaps  of  the  current  evaluation  method? 

a)  According  to  the  laws  of  statistics,  each  one  of  the  pair  wise  comparisons  includes 
risks  of  making  type  1  and  type  2  errors,  which  are  the  probabilities  of  making 
misjudgment  depending  on  a  statistical  comparison.  The  probability  of  making  a 
mistake  in  comparisons  increases  as  the  number  of  pairs  compared  increases. 

b)  According  to  the  Theory  of  Constraints;  local  improvements  do  not  necessarily 
mean  that  the  whole  system  improved  (Goldratt  and  Cox,  1992).  By  monitoring 
each  metric  individually,  we  may  not  visualize  the  big  picture  and  make  healthy 
decisions  at  a  strategic  level. 

c)  The  current  evaluation  method  may  be  useful  in  order  to  understand  if  the 
assessed  values  of  the  individual  metrics  satisfied  the  contractual  agreements. 

1  In  addition  to  these  two  processes,  some  experts  mentioned  that  in  some  cases  a  process  of  weighing  the  parameters  and  calculating  a 
sole  number  to  use  for  strategic  decisions  is  also  used.  According  to  our  research,  we  will  mention  this  additional  step  to  the  current 
evaluation  method  under  the  subject  heading  of  “The  Issues  about  the  Evaluation  Process  of  the  PBL  Systems”  at  the  end  of 
“Literature  Review”. 
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However,  as  some  academics  mentioned,  there  may  be  conflicts  between  the 
metrics  such  as:  “some  metrics  may  be  over  performing  while  the  others  are 
underperforming”  (Sols  et  ah,  2007). 

d)  “The  scarcest  resource  in  an  organization  is  a  manager’s  time.”  (Kaplan  and 
Cooper,.  1997,  p.144)  The  evaluation  of  the  metrics  individually  may  consume 
more  time  than  necessary  especially  for  the  senior  executives  for  two  reasons. 
First,  there  may  be  an  excessive  amount  of  defined  parameters  and  the  decision 
maker  may  need  time  to  understand  the  impact  of  each  one  of  them  on  the  system. 
Second,  there  may  be  conflicts  between  parameters.  Although  the  decision  maker 
may  spend  much  time  and  effort,  s/he  may  not  use  the  performance  data 
appropriately  in  the  decision-making  process.  (US  GAO  July  2008,  Highlights) 

Furthennore;  the  core  investigative  questions  that  will  shape  and  guide  this  study 
are  determined  as; 

4)  How  should  the  metrics  be  evaluated  to  improve  decision  making? 

5)  What  should  be  the  core  objectives  of  an  evaluation  model? 

6)  Can  the  proposed  model  in  this  thesis  be  used  for  forecasting,  or  building  new  PBL 
contracts? 

7)  At  which  decision  level  may  the  proposed  model  be  used? 

These  questions  will  be  answered  at  Chapter  V  according  to  the  results  gathered 
throughout  this  study. 
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Research  Focus 


The  reviewed  literature  and  the  knowledge  gathered  from  the  experts  suggest  that 
PBL  may  be  divided  into  three  parts  in  a  time  scale  (See  Figure  2).  This  research  is 
focused  on  the  post-  contract  period.  Second,  an  alternative  approach  to  monitor  the 
Perfonnance  Based  Logistics  (PBL)  supported  systems  both  at  the  individual  metrics 
level  and  at  the  system  level  is  sought,  so  the  evaluation  processes  and  approaches  are 
concentrated  on.  Finally,  a  model  is  built  in  order  to  fulfill  the  aim  of  proposing  an 
alternative  approach  in  this  area. 


Figure  2.  The  time  scale  of  PBL  applications 


time  scale 


Methodology 

First  of  all,  the  PBL  concept  was  studied  according  to  the  published  literature  and 
experts’  opinions.  Second,  gaps  in  the  PBL  literature  were  defined  in  order  to  make  a 
contribution  in  one  of  those  directions.  Third,  the  investigative  questions  were  created. 
Then  the  data  were  collected,  according  to  the  scope  of  this  study.  Next,  the  laws  of 
statistics,  calculus  and  geometry  were  used  to  process  the  gathered  data  and  built  an 
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integrated  model  to  evaluate  the  success  of  PBL  contracts.  Finally,  the  published 
literature  and/or  the  experts’  opinions  were  referenced  in  order  to  validate  the  model. 

Assumptions  /  limitations 

First  of  all,  the  research  is  concerned  with  the  post-  contract  period;  thus,  all  the 
parameters  were  defined,  negotiated  and  agreed  to  by  the  relevant  sides. 

Second,  all  the  performance  standards  relating  to  the  metrics  (both  qualitative  and 
quantitative)  were  mandated  with  the  time  frames  in  the  contract,  such  as:  80  %  of 
availability  at  the  end  of  every  operation  day,  reduction  of  logistic  footprint  at  the  end  of 
every  3  months,  or  85  %  user  satisfaction  at  the  end  of  every  month. 

Third,  all  the  assessed  numbers  of  the  metrics  must  satisfy  the  targeted  contractual 
numbers  or  in  other  words  the  bottom-line  constraints  individually  for  each  time  period. 

If  not  than  the  overall  perfonnance  of  the  contractor  must  satisfy  the  overall  contractual 
requirement  for  the  next  tenn. 

Organization  of  the  Thesis 

This  thesis  consists  of  five  chapters.  In  Chapter  I,  first  the  basic  problem  was 
defined  along  with  the  background  and  the  problem  statement.  Second,  the  research 
objectives  were  set  and  the  investigative  questions  created.  Third,  the  research  focus  was 
designated.  Then,  the  methodology  was  explained  with  the  relevant  assumptions  / 
limitations,  and,  finally,  with  the  organization  of  the  thesis,  the  map  of  the  thesis  was 
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revealed.  The  following  Chapters  both  extend  and  offer  a  more  detailed  study  about  the 
research. 

Chapter  II  presents  a  Literature  Review  about  the  PBL  concept,  including  its 
birth,  evolution  and  the  current  gaps  of  the  literature.  Chapter  III  describes  the  research 
methodology  of  this  thesis,  while  introducing  a  new  mathematical  model  about  the 
evaluation  of  the  metrics.  Chapter  IV  presents  the  results  and  checks  applicability  and  the 
usefulness  of  the  proposed  evaluation  process  model. 

Finally,  Chapter  V  presents  the  findings  and  recommendations,  as  well  as 
concluding  the  thesis  and  suggesting  future  studies  to  extend  the  research. 
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II.  Literature  Review 


Introduction 

In  any  kind  of  war,  the  participation  and  cooperation  of  the  whole  nation  is  the 
most  important  aspect  of  success.  The  reliance  on  contractors  increased  over  time  and  the 
ratio  of  the  number  of  contractors  over  the  number  of  troops  was  nearly  1 : 10  in  the  Gulf 
War  and  dramatically  increased  to  4:3  in  the  Balkans  between  1995  and  2000  (Vernon, 
2003).  Most  of  the  times,  the  contractors  seemed  to  operate  as  a  support  unit,  generally  in 

the  logistics  area,  for  the  armed  forces.  Although  these  are  not  the  clear  borders  in  a  time 

2 

scale,  military  logistics  can  be  separated  into  three  eras  for  the  United  States  (US). 

The  Traditional  Logistics  Era  continued  its  evolution  from  the  foundation  of  the 
US  to  the  end  of  the  Cold  War.  The  traditional  logistics  concepts  yield  predictive, 
optimized,  linear  supply  chains  that  operated  in  hierarchical  command  and  control 
structures  (Griffin,  2008). 

The  Transition  Logistics  Era  is  the  overlapping  part  which  started  since  the  end  of 
the  Cold  War.  Supply  Chain  Management  (SCM),  Sense  and  Respond  Logistics 
(S&RL)2 3,  Performance  Based  Logistics  (PBL)4  and  Enterprise  Resource  Planning  (ERP)5 
systems  and  networks  are  introduced  in  the  Department  of  Defense  (DoD).  New  ways  of 


2  The  separation  of  logistics  into  three  areas  for  the  US  is  constructed  by  the  researcher  of  this  study. 

3  The  S&RL  framework  exploits  advanced  technologies  through  highly  adaptive,  self-synchronizing  functional  processes  designed  to 
drive  shorter  decision  cycles  and  faster  responses. 

4  PBL  is  the  core  of  this  study  and  will  be  defined  clearly  in  the  next  chapters. 

5  The  ERP  systems  and  networks  are  being  applied  to  DoD  logistics  in  an  effort  to  create  a  connected  environmentin  which  near  real¬ 
time  data  can  be  exchanged  and  response  time  shortened. 
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thinking  such  as:  Total  Quality  Management  (TQM),  Lean,  the  Balanced  Score  Card 
(BSC),  Activity  Based  Costing  (ABC)  have  been  introduced. 

The  Integrated  Logistics  Era  is  the  future  goal  of  DoD.  This  era  defines  a  highly 
competitive  environment  between  the  contractors  and  a  highly  collaborative  environment 
between  the  contractors  and  the  Government.  The  main  expectation  is  answering  the 
intensively-elastic  needs  of  the  war  fighter  instantly  with  a  reasonable  cost  for  the  nation. 


The  History  of  PBL 


First  of  all,  it  may  be  proposed  that  the  PBL  concept  is  related  to  the  level  of  the 
technology  in  the  system.  The  US  The  Government’s  first  aircraft  contract  with  the 
Wright  brothers  in  1909  may  be  a  good  example  for  this  proposal  as  depicted  in  Figure  3. 


The  government  contracted  its  first 
aircraft  from  the  Wright  Brothers  in  1909 

•  Target  price:  $2S,000 

•  Target  speed:  40  MPH 

•  Incentive 

-  For  every  MPH  over  target,  contractor 
receives  additional  $2,500 

-  For  every  MPH  under  target,  contractor 
loses  $2,500 

-  Minimum  speed  requirement:  36  MPH 

-  Final  delivered  MPH  speed:  42  MPH 
•  Incentive  actually  earned:  $5,000 

Figure  3.  The  First  PBL  application  of  DoD 

(Gathered  from  DAU,  2008) 


13 


The  idea  behind  PBL  is  that  if  the  contractors  could  produce  a  highly 
technological  product  or  service,  then  they  should  know  how  to  improve  it  by  means  of 
reliability  and  cost  aspects.  So  limiting  them  by  mandating  how  to  produce  their  product 
or  service  under  non-PBL  contracts  may  result  in  losing  an  opportunity  of  bcnefitting 
from  their  innovative  approaches  and  cumulated  experience  (Rievley,  2001). 

After  the  Cold  War  period,  an  urgent  need  for  change  occurred  in  every  aspect  of 
the  armed  forces.  This  urgency  forced  the  DoD  to  transform  their  approach.  First  of  all,  a 
main  cause  of  the  collapse  of  the  Union  of  Soviet  Socialist  Republics  (USSR)  was  the 
misallocation  of  their  resources  and  inefficiencies  of  their  system.  Although  there  was 
one  enemy,  it  was  spread  out  geographically,  making  it  hard  for  the  USSR  to  carry  the 
heavy  burden  of  logistics  support  for  its  system.  History  clarifies  that,  using  a  large 
conventional  army  makes  war  very  expensive  and  requires  a  large  support  system  to 
maintain  effectiveness.  By  observing  the  enemy  well,  the  US  has  taken  the  necessary 
precautions  since  1987.  The  Depot  Maintenance  seemed  to  be  a  good  area  to  start, 
because  it  is  the  key  to  the  total  DoD  logistics  process  that  supports  millions  of 
equipment  items,  including  52,000  combat  vehicles,  350  ships  and  17,000  aircraft 
(Griffin,  2008).  For  the  US  DoD,  it’s  estimated  that;  80%  of  the  budget  is  used  for 
sustaining  the  defense  systems.  (Sols  et  ah,  2007)  As  it  is  mentioned  in  the  Government 
Accountability  Office  (US  GAO)  report  in  2001,  the  DoD  defined  three  series  of  actions 
primarily  to  shape  the  depot  Maintenance  environment.  The  actions  taken  and  the  results 
that  have  been  achieved  since  2001  are  described  below. 
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1)  The  base  realignment  and  closure  (BRAC)  process,  which  was  designed  to  reduce 
DOD’s  infrastructure;  has  reduced  DOD’s  cold-war— oriented  infrastructure  from  38 
military  depots  to  19  (See  Figure  4). 

2)  The  increased  reliance  on  the  private  sector  for  depot  maintenance  support  resulted  in 
the  contractors’  share  of  depot  maintenance  funding  increasing  by  90  percent  while 
the  military  depots’  share  of  funding  has  declined  by  6  percent  (See  Table  2). 

3)  The  major  downsizing  of  depot  maintenance  personnel  amounted  to  59  percent, 
which  was  the  third  highest  percent  of  any  category  of  DOD  civilian  personnel  (US 
GAO,  March  2001,  p.2). 


COMMISSION  ON 
ROLES  AND  MISSIONS 


BRAC  88 

A 


BASE  FORCE 
REVIEW 


BRAC  91  BRAC  93 

A_ A 


DEPOT  POLICY  REPORT  AND 
REVISED  REGULATIONS 


BRAC  95 

A  A 


QUADRENNIAL  DEFENSE 
REVIEW 

A 


1988  1989  1990  1991  1992  1993  1994  1995  1996  1997  1998  1999  2000  2001 


38  Major  Maintenance  Depots  19 

156.000  3  Civilian  Employees  I  64.500 

201 .6  Million  Direct  Labor  Hours  73.4  Million 


Figure  4.  Key  Events  Influencing  the  Reshaping  of  the  Military  Depot  System 

(Gathered  from  US  GAO,  March  2001) 
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Table  2.  Public  and  Private  Sector  Change  in  Depot  Maintenance  Funds 


(Gathered  from  US  GAO,  March  2001) 


Sector 

Fiscal  year 

1987 

(dollars  in  billions) 

Fiscal  year 

2000 

(dollars  in  billions) 

Change  in 
dollar  amount 

(dollars  in  billions) 

Percent  change 

Public 

$  8.7 

$8.2 

$-.5 

-6% 

Private 

$4.0 

$7.6 

$+3.6 

+90% 

Total 

$12.7 

$15.8 

+$3.1 

+24% 

By  the  terrorist  attacks  of  September  11,  2001  (9/1 1),  the  US  was  faced  with  a 
very  difficult  situation  of  shifting  from  traditional  anny  warfare  to  asymmetric  warfare. 
Actually,  the  concept  of  the  Global  War  on  Terrorism  (GWOT)  defined  by  the  US  is 
harder  than  what  the  USSR  experienced  in  the  Cold  War.  Briefly,  there’s  no  enemy 
country  but  there’s  an  enemy,  there  are  no  borders  of  the  conflict  zone,  but  everywhere  is 
potentially  a  conflict  zone  and  showing  a  presence  of  big  conventional  armed  forces  in  an 
area  is  no  more  a  show  of  superiority,  but  becoming  a  bigger  target  for  the  shadow 
enemy.  “The  new  threats  are  broader  and  include  global,  regional  and  local  elements. 
International  organizations,  allied  nation  states,  rogue  nations,  hostile  states  and  terrorist 
groups  all  contend  within  this  environment”  (Griffin,  2008). 

Dealing  with  terror  is  similar  with  the  children’s  Marco  Polo  game.  It  gets  the 
signals  of  the  presence  of  the  targets.  The  prevented  visibility  and  increased  mobility  of 
the  targets  makes  the  efforts  to  reach  them  meaningless  unless  any  of  the  targets  are 
caught  at  the  very  first  moment  with  a  reasonable  amount  of  effort.  The  aim  of  the  targets 
is  taking  as  much  energy  as  possible  to  make  the  enemy  exhausted.  As  seen  from  a  very 
simple  child  game,  allocation  of  resources  and/or  directing  the  opponent  to  misallocate  its 
resources  is  the  key  to  success  for  everyone. 
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The  budget  dynamics  of  the  terrorist  organizations  are  much  different  than  the 


Governments,  which  make  them  more  resilient  and  harder  to  deal  with.  An  example  can 
be  given  of  one  income  item,  the  drug  trafficking,  for  a  terrorist  organization  that  the  US 
deals  with.  According  to  the  recently  published  two  reports  of  the  United  Nations  (UN), 
92%  of  the  global  opium  production  is  at  the  southern  regions  of  Afghanistan  in  the  year 
2006.  This  raw  material  for  heroin  soared  in  2007,  rising  34  percent  from  the  already 
record  levels  of  2006,  under  the  control  of  the  Taliban  insurgency  (UN  DRUG  REPORT, 
2007;  UN  DRUG  REPORT,  2008). 

Not  only  the  production,  but  also  the  transportation  and  the  distribution  activities 
are  governed  by  a  network  of  terrorist  organizations.  In  2007,  Afghanistan  either 
processed  and  exported  or  stored  for  future  export  some  660  tons  of  heroin  and  morphine. 
Because  of  the  risks  associated  with  smuggling,  opium  gains  value  at  every  border  it 
crosses.  By  the  time  heroin  reaches  its  final  destination,  its  price  may  have  gone  up  100 
times  and  the  reports  found  indications  that  the  Taliban  may  have  started  to  use  drug 
profits  to  purchase  arms,  pay  militia  forces  and  finance  logistics  (UN  DRUG  REPORT, 
2008). 

Because  of  some  reasons  explained  above,  GWOT  most  recently  referred  to  as  the 
long  war,  has  stimulated  all  US  federal  agencies  to  review  their  ongoing  operations  and 
budget  to  concentrate  on  cost  savings. 

However,  concentrating  on  cost  savings  without  understanding  the  demanding 
nature  of  asymmetric  warfare  includes  a  big  risk.  According  to  the  Mid-  Session  Review 
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of  the  US  Budget6  2009,  the  President  is  mentioned  as  “committed  to  restrain  Federal 
spending  for  the  purpose  of  balancing  the  budget  by  2012”  (EOP  US  Mid-Session 
Report,  July  2008,  p  3).  However,  in  the  same  document,  the  2008  Supplemental 
Appropriations  Act  is  mentioned.  According  to  that  act,  U.S.  troops  in  Iraq  and 
Afghanistan  will  be  provided  with  the  resources  they  need  to  fight  the  Global  War  on 
Terror  and  the  cost  savings  are  targeted  on  non-security  discretionary  spending.  The 
objectives  are  mentioned  as; 

elimination  or  reducement  of  15 1  unnecessary  or  duplicative  programs  totaling 
$  1 8  billion  and  holding  the  growth  in  non-security  spending  to  below  one  percent 
for  2009  (EOP  US  Mid-Session  Report,  July  2008,  p  3). 

In  addition  to  the  President’s  targets,  the  cost  saving  opportunities  via  the 

improvements  of  Supply  Chain  Management  (SCM)  activities  may  also  be  mentioned. 

According  to  the  revised  budget  of  the  US  for  fiscal  year  2009,  the  share  of  the  total 

Defense  sector  is  requested  as  $515.4  billion  and  the  requested  DoD  Budget  is  $17,647 

billion.  (EOP  US  Mid-Session  Report,  July  2008,  p  23)  Assuming  a  very  conservative 

target  of  1%  of  cost  savings  via  improvements  in  SCM  activities  throughout  the  Defense 

sector,  this  would  create  an  extra  resource  of  $5,154  billion. 

Just  3  weeks  after  9/11,  the  2001  Quadrennial  Defense  Review7  (OSD  QDR)  was 

published  and  compressing  the  supply  chain  via  modernizing  business  processes  and 

using  Perfonnance  Base  Logistics  (PBL)  are  mentioned  as  logistics  transfonnations 


6  The  Mid-Session  Review  is  the  supplemental  updated  form  of  the  US  budget  by  the  President.  It  contains  revised  estimates  of 
receipts,  outlays,  budget  authority  and  the  budget  deficit  or  the  surplus  for  fiscal  years  2008  through  2013. 

7  QDRs  are  conducted  every  four  years  to  ascertain  what  the  military  will  need  in  manpower  and  materiel  to  address  anticipated 
missions  envisioned  20  years  in  the  future. 
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initiatives  in  the  report  (Mahon,  2007).  Basically,  the  OSD  QDR  was  declaring  the 
fundamentals  of  the  new  era  of  logistics  as: 


To  ensure  the  Department  transforms  its  logistics  capabilities,  DoD  will  pursue 
actions  to  sustain  the  force  more  effectively  and  efficiently.  Specific  areas  will 
include  a  dramatically  improved  deployment  process  and  accelerated 
implementation  of  logistics  decision  support  tools.  DoD  must  also  accelerate 
logistics  enterprise  integration,  reduce  logistics  demand  and  reduce  the  cost  of 
logistics.  In  addition,  conducting  industrial  vulnerability  assessments  and 
developing  sustainment  plans  for  the  most  critical  weapons  systems  and  preferred 
munitions  will  help  ensure  effective  sustainment.  (OSD  QDR,  2001,  p  35) 

In  addition,  the  Chainnan  of  the  Joint  Chiefs  of  Staff  mentioned  the  need  of  an 

integrated  system  in  his  statement  by: 

If  truly  dramatic  improvement  in  future  joint  operational  effectiveness  is  to  be 
achieved,  however,  more  is  required.  First,  a  DOD-wide  transfonnation  strategy,  a 
joint  organizing  vision  and  a  joint  transformation  roadmap  are  essential  to  guide, 
integrate  and  synchronize  the  efforts  of  the  Services.  Second,  we  need  DOD-wide 
refonn  of  key  institutional  planning,  programming,  budgeting  and  acquisition 
processes.  These  two  requirements  are  interdependent;  no  real  progress  will  be 
made  in  one  without  the  other.  (OSD  QDR,  2001,  p.68) 

Meanwhile  it  was  projected  for  2001  that  protecting  and  advancing  US  interests 

around  the  world  involve  over  $  1  trillion  in  assets,  about  3  million  military  and  civilian 

employees  and  about  an  annual  budget  of  $310  billion  then  (US  GAO.,  January  2001,  p 

6). 


In  2002,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD  AT&L)  mandated  the  services  to  develop  and  implement  a  schedule  to 
aggressively  apply  Performance  Based  Logistics  (PBL)  to  all  new  weapons  systems,  as 
well  as  all  Acquisition  Category  (ACAT)  I  and  II  legacy  systems  before  May  1,  2002 
(Mahon,  2007). 
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In  2003  the  DoD  Directive  5000.1  which  is  also  known  as  the  Defense 


Acquisition  System  (DAS)  was  implemented  and  performance-based  strategies  for 
procurement  and  sustainment  of  the  products  and  services  were  mentioned  as  “to  do 
whenever  feasible”  (DoD  Directive  5000.1,  2003,  p  9). 

In  addition  to  that,  “The  Primary  Objective  of  Defense  Acquisition”  is  defined  as: 

to  acquire  quality  products  that  satisfy  user  needs  with  measurable  improvements 
to  mission  capability  and  operational  support,  in  a  timely  manner  and  at  a  fair  and 
reasonable  price.  (DoD  Directive  5000.1,  2003,  p  6) 

Although  implementation  of  PBL  was  pointed  as  a  target,  it  was  not  described  to 
Program  Managers  (PMs)  how  to  implement  it.  The  ambiguity  of  the  changeover  period 
and  the  unanswered  questions  in  the  minds  of  acquisition  professionals  cumulatively 
increased.  Then  the  USD  AT&L  tried  to  eliminate  the  confusion  by  two  publications  in 
2003.  The  first  one  was  listing  key  activities  and  outputs  which  PMs  should  consider 

when  assessing  a  program  for  a  PBL  strategy  and  the  second  one  was  the  targets  for  the 

8 

percentage  of  contract  dollars  awarded  using  performance-based  service  activities 
(Mahon,  2007).  The  Office  of  Secretary  of  the  Defense  (OSD)  joined  the  description 
process  by  publishing  the  Designing  and  Assessing  Supportability  in  DoD  Weapons 
Systems:  A  Guide  to  Increased  Reliability  and  Reduced  Logistics  Footprint  which  is  also 
referred  to  as  the  Supportability  Guide  (SG)  in  October  2003.  The  SG  tried  to  address  and 
explain  the  responsibilities  of  PMs  in  the  Total  Life  Cycle  Systems  Management 
(TLCSM)  concept,  which  is  briefly  the  responsibility  for  a  system  from  cradle  to  grave 
(OSD,  2003).  The  SG  filled  up  a  big  gap  about  the  implementation  of  PBL  in  addition  to 


8  The  targets  were  set  as:  %25  for  Fiscal  Year  (FY)  2003,  %35  for  FY  2004,  %50  for  FY  2005. 
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DoD  Directive  5000.1,  2003,  the  Defense  Acquisition  System  (DAS)  and  DoD 

Instruction  5000.2,  the  Operation  of  the  DAS.  In  the  SG  the  necessity  of  integration  of 

performance  and  the  other  elements  of  effectiveness  is  mentioned  as: 

Performance  cannot  be  considered  separate  from  the  other  elements  of  operational 
effectiveness  -  they  are  inextricably  linked.  The  system  capabilities  and  functions 
represent  the  desired  mission  capabilities  as  a  total  package,  together  with  the 
sustainment  objectives  and  the  desired  logistics  footprint  reductions.  As  discussed 
in  the  following  paragraph,  in  the  current  operational  and  budgetary  context, 
priorities  must  be  complemented  with  an  emphasis  on  system  availability  — 
Reliability,  Maintainability,  Supportability  (RMS)  and  producibility  (OSD  ,2003, 

pin. 

Meanwhile  new  issues  arose  concerned  with  the  PBL  implementations  of  PMs 
and  the  office  of  the  USD  AT&L  continued  their  publications  in  January  2004  and  March 
2004.  This  continuation  of  policy  and  guidance  releases  suggests  that  a  major 
transformation  in  product  support  may  have  been  prematurely  implemented  by  DoD 
(Mahon,  2007) 

Finally,  the  Defense  Acquisition  University  (DAU)  explained  the  DoD  policy  in 
their  publications;  “Performance  Based  Logistics:  A  Program  Manager's  Product  Support 
Guide”,  which  is  also  referred  as  the  PBL  Guide9,  on  March  2005  (DAU,  2005). 
According  to  that  publication: 

Performance  Based  Logistics  (PBL)  is  the  purchase  of  support  as  an  integrated, 
affordable,  perfonnance  package  designed  to  optimize  system  readiness  and  meet 
performance  goals  for  a  weapons  system  through  long-term  support  arrangements 
with  clear  lines  of  authority  and  responsibility.  Simply  put,  perfonnance  based 
strategies  buy  outcomes,  not  products  or  services  (DAU,  2005,  p  1-1). 

PMs  optimize  performance  and  cost  objectives  by  the  implementation  of  PBL  in 

varying  degrees  of  The  Government-industry  partnerships  as  depicted  in  Figure  5. 


9  The  DAU’s  publication  “Performance  Based  Logistics:  A  Program  Manager's  Product  Support  Guide”  on  March  2005  was  referred 
to  as  the  PBL  Guide  in  the  literature  then. 
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More  Organic 


More  Commercial 


Organic  providers  Public-private  Contractors 

responsible  for  partnering  responsible  for 

majority  of  support  opportunities  majority  of  support 


Figure  5.  The  Government-industry  partnerships 

(Gathered  from  DAU,  2008) 


Before  explaining  the  implementation  of  PBL,  it  would  be  useful  to  mention  some 
key  components  of  PBL  which  are:  the  Integrated  Product  Teams,  PBL  metrics, 
Performance  Based  Agreements  and  Business  Cases  Analyses 


Integrated  Product  Teams 

PBL  Integrated  Product  Teams  (IPTs)  which  are  also  referred  to  as  the  main 
stakeholders  of  implemented  PBL  programs,  are  war  fighters,  users,  developers, 
acquirers,  technologists,  testers,  budgeters  and  sustainers  experts  and  war  fighter 
representatives(DoDD  5000.1,  2003,  para.  El.1.2.).  A  sample  PBL  IPT  team  depicted  in 
Figure  6. 
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Figure  6.  A  Sample  PBL  IPT 

(Gathered  from  DAU,  2005) 


This  team  should  work  together  from  the  beginning  of  the  program  and  it  is 


essential  for  the  success  of  any  program  that  members  are  able  to  work  across 


organizational  boundaries. 


PBL  Metrics 

An  effective  PBL  implementation  depends  on  metrics  that  accurately  reflect  the 
user’s  needs  and  can  be  effective  measures  of  the  support  provider’s  performance.  The 
PBL  Top-Level  Metrics  are  defined  in  the  DAU  (2005)  as: 

1)  “Operational  Availability  (Ao)  is  the  percent  of  time  that  a  system  is  available  for  a 
mission  or  the  ability  to  sustain  operations  tempo”  (DAU,  2005,  p.  2-5). 
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2)  “Operational  Reliability  (R)  is  the  measure  of  a  system  in  meeting  mission  success 
objectives  (percent  of  objectives  met,  by  system)”  (DAU,  2005,  p.  2-5). 

3)  Cost  per  Unit  Usage  (CUU)  is  the  total  operating  cost  divided  by  the  appropriate  unit 
of  measurement  for  a  given  system”  (DAU,  2005,  p.  2-5). 

4)  Logistics  Footprint  (LF)  is  the  Government/contractor  size  or  “presence”  of  deployed 
logistics  support  required  to  deploy,  sustain  and  move  a  system.  Measurable  elements 
include  inventory/equipment,  personnel,  facilities,  transportation  assets  and  real 
estate”  (DAU,  2005,  p.  2-5). 

5)  Logistics  Response  Time  (LRT)  is  the  period  of  time  from  logistics  demand  signal 
sent  to  satisfaction  of  that  logistics  demand.  “Logistics  demand”  refers  to  systems, 
components,  or  resources  (including  labor)  required  for  system  logistics  support” 
(DAU,  2005,  p.  2-6). 

However,  these  five  metrics  defined  by  the  PBL  Guide  may  not  be  appropriate  to 
use  raw  for  every  PBL  contract.  First  of  all,  these  metrics  are  general  and  composite,  for 
example:  Operational  Availability  (Ao),  may  be  defined  as  the  ratio  of  mean  time 
between  maintenance  (MTBM)10  to  total  system  time  (Kumar  et  ah,  2007). 

Thus,  each  program  should  consider  and  tailor  some  or  all  of  these  metrics,  as 
well  as  define  new  metrics  for  their  program.  Ledyard  and  Vitasek  (2004)  offered  five 
rules  that  may  be  used  for  the  measurement  process  as  depicted  in  Figure  7. 


10  According  to  our  literature  review.  Mean  Time  Between  Failure  (MTBF)  is  also  used  instead  of  MTBM. 
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Process  Checle 


Figure  7.  Five  Rules  Approach  for  Good  Measurement 

(Gathered  from  Ledyard  and  Vitasek,  2004) 

Performance  Based  Agreements  and  Business  Case  Analyses 

Before  discussing  Performance  Based  Logistics  (PBL)  implementation,  it  is 
important  to  mention  two  key  PBL  implementation  issues,  which  are  Performance  Based 
Agreements  (PBAs)  and  PBL  Business  Case  Analyses  (PBL  BCAs). 

PBAs  are  one  of  the  key  components  of  an  effective  product  support  strategy. 
(DoDD  5000.1,  2003,  para  El.1.16.)  They  establish  the  negotiated  baseline  of 
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performance  and  corresponding  support  necessary  to  achieve  that  performance,  whether 
provided  by  commercial  or  organic  support  providers. 

According  to  the  PBL  Guide  BCAs  should  consider: 

•  The  relative  cost  versus  benefits  of  different  support  strategies 

•  The  methods  and  rationale  used  to  quantify  benefits  and  costs 

•  The  impact  and  value  of  performance/cost/schedule/sustainment  trade-offs 

•  Data  required  to  support  and  justify  the  PBL  strategy 

•  Sensitivity  of  the  data  to  change 

•  Analysis  and  classification  of  risks 

•  A  recommendation  and  summary  of  the  implementation  plan  for  proceeding 
with  the  best  value  alternative  (DAU,  2005,  p  3-28). 


Implementation  of  PBL 

The  PBL  implementation  process  is  flexible  and  PBLs  differ  in  scale,  covering  a 
broad  range  from  component  level  up  to  system  platform  level.  On  one  hand,  DAU 
academics  proposed  in  the  PBL  Guide,  that  the  12 -step  process  to  implement  a  PBL 
program  into  consideration  be  tailored  appropriately  to  the  level  of  the  system.  (See 
Figure  8) 


Figure  8.  PBL  Implementation  steps 

(Gathered  from  DAU,  2005) 
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On  the  other  hand  Berkowitz  et  al.  (2005)  offered  six  guidelines  briefly  explained 

below: 

1)  “Assign  responsibilities  clearly  throughout  the  firm —  Blanket  statements  about 
policy  changes  that  imply  that  ‘PBL  is  everyone’s  responsibility’  are  typically 
ineffective”  (Berkowitz  et  ah,  2005,  pp  263-264). 

2)  “Design  metrics  to  motivate  the  right  behavior —  Metrics  designed  to  motivate  the 
right  behavior  must  be  carefully  crafted  and  applicable  across  the  entire  organization” 
(Berkowitz  et  ah,  2005,  pp  263-264). 

3)  “Manage  failures  to  limit  disincentives  for  risk-taking —  Failure  is  part  of  the 
learning  process”  (Berkowitz  et  al.,  2005,  pp  263-264). 

4)  “Develop  a  supportive  organizational  context  for  tools —  These  tools  include 
middleware  to  standardize  decision-making  based  on  legacy  system  output  and 
tracking  systems  to  document  performance  improvements  and  lessons  learned  across 
the  organization”  (Berkowitz  et  al.,  2005,  pp  263-264). 

5)  “Manage  relationships  with  stakeholders —  Continuing  communication  with 
stakeholders  is  one  way  to  gain  their  support”  (Berkowitz  et  al.,  2005,  pp  263-264). 

6)  “Benchmark  to  promote  continuous  improvement —  In  order  to  find  how  well 
initiatives  are  working,  compare  results  through  the  benchmarking  process 
(Berkowitz  et  al.,  2005,  pp  263-264). 
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Risk  vs.  Contract  Types  in  PBL 


Buying  performance  instead  of  materials  and/or  services  includes  risk  implications 
both  for  the  contractor  and  the  Government.  Sols  et  al.  (2007)  proposed  a  reasonable 
relationship  between  risk  and  contract  types  as  depicted  in  Figure  9. 

General  contract  types  of  PBL  mentioned  in  Sols  et  al.  (2007)  are  described  in  the 
Federal  Acquisition  Regulations  (FAR)  as: 

1)  “Finn  Fixed  Price  (FFP):  A  firm- fixed-price  contract  provides  for  a  price  that  is  not 
subject  to  any  adjustment  on  the  basis  of  the  contractor’s  cost  experience  in 
performing  the  contract”  (GSA,  2005,  p  16.2-1). 

2)  “Cost  Plus  Fixed  Fee  (CPFF):  A  cost-plus-fixed-fee  contract  is  a  cost  reimbursement 
contract  that  provides  for  payment  to  the  contractor  of  a  negotiated  fee  that  is  fixed  at 
the  inception  of  the  contract”  (GSA,  2005,  p  16.3-1). 

3)  “Fixed  Price  Incentive  (FPI):  A  fixed-price  incentive  (finn  target)  contract  specifies  a 
target  cost,  a  target  profit,  a  price  ceiling  (but  not  a  profit  ceiling  or  floor)  and  a  profit 
adjustment  formula”  (GSA,  2005,  p  16.4-2). 

4)  “Cost  Plus  Incentive  Fee  (CPIF):  The  cost-plus-incentive-fee  contract  is  a  cost- 
reimbursement  contract  that  provides  for  the  initially  negotiated  fee  to  be  adjusted 
later  by  a  formula  based  on  the  relationship  of  total  allowable  costs  to  total  target 
costs”  (GSA,  2005,  p  16.4-3). 


28 


Fiim  Fixed  Price  (FFP) 


Most  efficient  for  client  when 
uncertainty  is  low  and  risks  are 
controllable  by  contractor. 


Not  appropriate  when 
uncertainties  are  high. 


Cost  Plus  Fixed  Fee  (CPFF) 


Most  efficient  for  client  when 
uncertainty  is  high. 


Not  appropriate  when 
uncertainties  are  high.  High 
risk  to  client  of  inefficient 
contractor  management 


cost  Plus  incentive  Fee  (CPIF) 

Risks  are  shared  in  light  of 
uncertainties.  Risk-sharing  is  based 

on  relative  levels  of  risk  aversion  by 

client  and  contractor.  In  general 
client  takes  higher  risk. 

Neither  party  has  fulfillment  of  own 

goals  is  totally  ensured. 


Fixed  Price  Incentive  (FPI) 

Risks  are  shared  in  light  of 
uncertainties.  Risk-sharing  is  based 

on  relative  levels  of  risk  aversion  by 

client  and  contractor.  In  general 
contractor  takes  higher  risk. _ 

Neither  party  has  fulfillment  of  own 
goals  is  totally  ensured. 


Performance  Based  Logistics 
Support  (PBLS) 


Client  has  maximum  assurance  of 

system  performance  and  achievement 
of  operational  and  support  goals,  and 
contractor  has  highest  motivation  to 
perform  and  earn  fair  profit. 


Contract  negotiation  and  follow-up 
require  considerable  resources  to 
both  parties. 


Increasing  client  assurance 
of  achievement  of  goals, 
and  contractor  assurance 

of  earning  of  fair  profit. 

Figure  9.  Contract  Types  vs  Risk 

(Gathered  from  Sols  et  al.,  2007) 


The  Gaps  in  the  PBL  Literature 

PBL  is  a  relatively  new  concept  and  there  are  still  some  important  issues  that 
should  be  addressed.  According  to  the  reviewed  literature,  the  main  issues  about  PBL 
may  be  grouped  under  three  main  headings  which  are:  the  Ethical  Issues  of  Reliance  of 
Contractors  in  Battlefields,  the  Issues  about  Contract  Length,  and  the  Measurement 
Issues  of  the  PBL  Systems. 

The  Ethical  Issues  of  Reliance  of  Contractors  in  the  Battlefield 

Openshaw  (2006)  addressed  some  important  ethical  issues  related  to  contractor 
reliance  in  his  study.  Briefly,  the  contractors  are  not  military  members,  so  basically  do 
not  have  the  same  status  as  a  military  soldier.  Under  the  international  agreements 
between  the  Governments,  military  members  are  mentioned;  however  the  contractors  are 
exempted  in  many  cases.  Because  of  this  reason,  in  the  chaotic  atmosphere  of  the  conflict 
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zones,  actions  and  the  contingencies  related  with  the  actions  of  the  contractors  may  cause 
more  damage  than  the  benefit  for  the  Governments.  For  a  worst  case  scenario,  contractors 
may  resign  at  any  time  and  leave  the  combat  zone,  which  would  paralyze  the  capability 
of  the  troops  almost  immediately  (Openshaw,  2006). 

Issues  about  Contract  Length 

Generally,  PBL  contracts  create  a  necessity  for  contractors  to  make  upfront 
investments.  That’s  why  longer  contract  tenns  allow  contractors  to  spread  the  investment 
costs  over  a  larger  base  and  this  may  create  a  long-term  relationship  between  the 
Government  and  the  contractor  (Griffin,  2008).  On  the  other  hand,  this  approach 
includes  some  risks  for  the  Government.  Gardner  (2008)  defines  some  of  the  potential 
risks  as:  “supplier  opportunism,  selection  of  the  wrong  supplier,  supplier  volume 
uncertainty,  foregoing  to  other  businesses  of  suppliers  and  unreasonableness  of  buyers” 
(Gardner,  2008). 

Measurement  Issues  of  the  PBL  Systems 

The  PBL  concept  was  created  to  achieve  a  higher  level  of  outcome  by  using  the 
capabilities  of  the  contractors  such  as  business  experience,  innovation  and  technical 
background  (Rievley,  2001).  However,  before  giving  a  complete  authority  to  the 
contractor  (supplier)  about  how  they  would  do  something,  the  Government  (customer) 
should  be  sure  of  what  it  would  get.  Thus,  the  desired  outcomes  would  be  defined  and 
then  appropriate  perfonnance  metrics  would  be  developed  for  being  sure  of  the  desired 
outcomes.  However,  there  are  gaps  between  the  war  fighter  -oriented  The  Government 
metrics  and  the  profit  -oriented  contractor  metrics  and  how  the  services  provided  by  the 
contractors  would  be  measured  by  means  of  The  Government  objectives.  For  example, 
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reducing  the  logistics  foot-print  (LFP)  may  be  a  desired  outcome  for  the  Government,  but 
it  is  hard  to  translate,  quantify  and  measure.  The  contractors  may  not  perceive  it  the  same 
way  as  the  Government  and  higher  transaction  costs  could  be  the  result  (Doerr  et  ah, 
2005). 

According  to  some  academics,  the  PM’s  approach  should  be  pragmatic  while 
establishing  the  metrics  (Ledyard  and  Vitasek  2004).  However,  some  earlier  academics 
called  attention  to  the  trap  of  “if  you  can’t  measure  what  you  want,  want  what  you  can 
measure”  (Kaplan  and  Cooper,  1996,  p  100).  Instead,  they  insisted  on  searching  new 
approaches  in  order  to  develop  new  metrics  and  new  measurement  approaches  to  achieve 
the  desired  outcomes  (Kaplan  and  Cooper,  1996,  p  100)  .The  National  Aeronautics  and 
Space  Administration  (NASA)  Headquarters  Performance  Based  Contracting  Assessment 
of  1999  confirms  this  by  mentioning  the  difficulties  in  developing  appropriate  and 
meaningful  perfonnance  standards  for  advanced  Research  and  Development  (R&D) 
activities  (Rievley,  2001).  Besides,  this  doesn’t  mean  that  they  should  give  up  searching 
to  develop  new  metrics  and  demanding  perfonnance  outcomes  that  are  vitally  important 
for  critical  mission  accomplishments. 

Generally  achieving  the  desired  levels  of  reliability  and  availability  at  a 
reasonable  cost  is  a  common  expectation  from  the  contractor  (Smith,  2004).  But  this 
expectation  covers  a  wide  range.  Three  main  measurement  issues  related  with  PBL 
within  the  scope  of  this  research  are: 

1)  Defining  desired  levels  of  perfonnance:  Kaplan  and  Norton  (1996)  offered  an 
integrated  approach  from  defining  the  mission  of  the  company  to  constructing  an 
entire  organization  around  that  mission  statement  in  the  Balanced  Score  Card  (BSC). 
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Basically,  the  aim  of  the  BSC  is  deriving  the  mission  of  the  company  and  achieving 
the  strategy  that  would  lead  the  company  to  reach  that  goal  and  then  deriving  the 
strategy  in  order  to  build  relevant  metrics  that  would  translate  “the  mission  and 
strategy  into  action”  (Kaplan  and  Cooper,  1996).  They  defined  four  sets  of 
performance  metrics  perspectives  that  are:  financial,  customers,  internal  business 
processes  and  learning  and  growth  (Kaplan  and  Cooper,  1996).  Three  important 
aspects  of  the  BSC  for  the  current  study  are: 

a)  The  performance  metrics  are  mentioned  and  developed  according  to  the  defined 
mission  and  strategy  by  four  main  dimensions  and  many  sub-dimensions. 

b)  All  the  defined  metrics  should  have  cause  and  effect  relationships  by  means  of 
whether  or  not  serving  to  the  strategy.  These  cause  and  effect  relationships  define 
the  hypothesized  correlations  among  the  metrics. 

c)  “Causal  paths  that  form  all  the  measures  on  a  score  card  should  be  linked  to 
financial  objectives”  (Kaplan  and  Cooper,  1996,  p  151).  On  the  Government  side 
financial  measures  are  cost  savings  and  budget  constraints. 

Two  important  issues  of  the  BSC  are  relevant  to  the  current  study.  First,  their 
opening  argument  in  the  BSC  criticizes  the  usage  of  a  single  metric  in  order  to 
monitor  the  system  via  an  airplane-pilot  example  that  mentions  the  existence  of  many 
different  flight  instruments  (Kaplan  and  Cooper,  1996,  pp  1-2).  Although  accepting 
this  idea,  we  also  may  call  attention  to  the  cockpit  windows,  or  canopies  of  the 
aircrafts  in  addition  to  many  other  instruments.  By  the  existence  of  the  windows,  the 
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pilot  can  evaluate  the  current  overall  position  of  the  aircraft  in  a  short  time  frame  and 
then  may  check  the  relevant  instruments  in  order  to  take  the  necessary  steps. 

Second,  linking  of  all  the  measures  to  the  financial  outcomes  in  BSC  may  be 
appropriate  for  commercial  organizations;  however,  financial  outcomes  may  not  be 
the  only  desired  outcomes  in  PBL-governed  weapon  systems  for  the  Government.  In 
other  words,  the  desired  outcomes  for  the  Government  may  include  some  additional 
areas  and  the  metrics  that  are  related  to  different  specific  areas  may  conflict  with  each 
other.  Therefore,  a  higher  level  sight  may  be  needed  to  govern  the  system. 

In  conclusion,  the  BSC  is  a  necessary  mindset  while  building  the  metrics  and  the 
measurement  and/or  management  system.  Although  constructing  a  BSC  takes  time 
and  effort,  a  strategic-level  BSC  should  be  built  for  the  organization  and  the  relevant 
metrics  should  be  set  before  starting  to  negotiate  the  PBL  contract  for  a  weapon 
system. 

2)  Cost  Management:  Achievable  cost  reductions  are  expected  from  PBL-govemed 
systems  and  Business  Case  Analyses  (BCAs)11  are  very  important  in  the  PBL 
implementation  process.  However,  in  some  situations,  cost  data  neither  exists  nor  is 
accurately  allocated  (Rievley,  2001;  US  GAO,  2005,  p  7;  Mahon,  2007).  According 
to  the  reviewed  literature,  Activity  Based  Costing  (ABC)  may  be  the  right  choice  for 
PBL-govemed  systems.  In  addition  to  that,  Traditional  Cost  Accounting  (TCA) 
systems  are  not  appropriate,  but  mandated  because  of  financial  reporting  purposes: 
especially  for  tax  issues  for  the  commercial  companies  (Kaplan  and  Cooper,  1997,  p 
13).  The  Government  also  relies  on  TCA  systems  for  funding  and  spending  issues. 

11  See  pp  21-22  in  this  study  for  a  more  detailed  description  of  BCAs. 
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Activity  Based  Costing  (ABC)  is  an  approach  especially  for  the  service  industries 
in  order  to  link  the  costs  of  the  resources  they  supply  to  the  outcomes  gathered  by  the 
service  provided  (Kaplan  and  Cooper,  1997,  p  228).  The  most  important  aspects  of 
ABC  related  with  the  current  study  may  be  mentioned  as: 

a)  It  is  an  integrated  approach  similar  to  the  BSC,  which  concentrates  on  linking  the 
cost  of  supplies  used  with  the  outcomes  gathered.  Allocating  the  costs  according 
to  the  cause  and  effect  relationships  among  the  different  levels  of  the  system 
makes  ABC  potentially  more  accurate  than  TCA  systems. 

b)  Parallel  to  the  usage  of  ABC  systems,  accurate  cost  data  may  be  gathered  in  order 
to  build  a  BCA. 

c)  Some  important  cost  metrics  may  be  developed  via  the  usage  of  ABC. 

Evaluation  Processes  of  PBL 

Each  system  has  some  inputs  and  some  outcomes.  The  objective  of  the  PBL  is 
minimizing  the  committed  resources  (inputs)  while  maximizing  the  achieved  levels  of 
performance  outcomes  and  the  mission  of  the  evaluation  processes  is  to  ensure  that  the 
system  is  serving  the  objective.  Generally  problems  occur  because  of  either  the 
inappropriately  mandated  perfonnance  goals,  or  misunderstood  metrics  (Sols  et  ah, 

2007). 

Kumar  et  al  (2007)  offered  an  optimization  model  to  both  The  Government  and 
the  contractor  sides.  In  their  study,  it’s  proposed  that  the  contractors  would  optimize 
multiple  performance  measures  of  their  product  by  choosing  the  optimum  sub-units  at  the 
design  level  and  the  Government  officials  would  choose  the  best  design  between 
competing  firms  via  the  usage  of  their  model  (Kumar  et  al.,  2007).  Despite  being  out  of 
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the  scope  of  the  current  study  ,  it  is  important  to  mention  some  gaps  of  their  model. 

First,  the  model  is  a  descriptive  one  and  some  elements  such  as  annual  failure  rate  of  the 
parts,  stock  levels  of  parts,  annual  repair  rates  are  assumed  to  be  known  between 
alternatives  before  the  design  stage.  However,  these  elements  may  only  be  estimated  with 
related  variances  in  the  best  case.  Second,  the  logistics  footprint  is  defined  as  a  decision 
parameter  for  their  model,  even  though  it  was  not  explained  how  it  was  calculated  as  a 

1  T 

sole  number  .  Third,  as  they  mentioned  in  their  study,  the  increase  of  the  number  of 
components  and  the  subsystems  would  make  it  impossible  to  solve  with  an  exact 
algorithm. 

The  common  sense  in  implementing  a  PBL  contract  is  to  incentivize  the  contractor  to 
achieve  the  desired  levels  of  performance.  Some  academics  suggested  defining  dead 
(neither  penalty  nor  reward),  penalty  and  bonus  zones  for  each  one  of  the  figures  of 
merits  (FOMs)14,  compensation  between  pairs  of  metrics,  and  proposed  a  multi¬ 
dimensional  dead,  penalty,  and  reward  zones  approach  (Sols  et  ah,  2007).  These  concepts 
are  depicted  in  Figures  10,  1 1  and  12. 


12  The  optimization  model  mentioned  in  the  study  of  Kumar  et  al.,  (2007),  is  mainly  concerned  with  the  Pre-  PBL  period. 

(See  Figure  3) 

ljThe  Logistic  Footprint  includes  inventory/equipment,  personnel,  facilities,  transportation  assets  and  real  estate  by  definition, 
assuming  it  as  a  sole  number  without  showing  how  it  was  calculated  in  a  model  may  be  an  over  assumption. 

14  The  word  Figure  of  Merit  (FOM)  is  used  instead  of  metrics  in  Sols  et  al.,  (2007) 
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Figure  10.  Dead,  Bonus  and  Penalty  Zones  Concept 

(Gathered  from  Sols  et  ah,  2007) 


Figure  11.  3-Dimensional  Dead,  Penalty  and  Reward  Zones 

(Gathered  from  Sols  et  ah,  2007) 
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FOM,L  FOM,u  FOM, 


Figure  12.  3-Dimensional  Dead,  Penalty  and  Reward  Zones15 

(Gathered  from  Sols  et  al.,  2007) 


The  multi-dimensional  dead  (neither  penalty  nor  reward),  penalty  and  reward 
zones  approach  includes  some  important  aspects  for  the  current  study.  First,  the  idea  of 
linking  two  metrics  in  a  3-dimensional  shape  and  defining  9  possible  zones  for  the  two 
metrics  reveals  the  difficulty  of  integrating  metrics  in  a  model.  Second,  each  one  of  the 
metrics  is  defined  with  a  variance,  so  the  model  is  prescriptive.  Third,  compensation 
between  pairs  of  metrics  is  introduced.  This  idea  with  the  usage  of  weighted  average 
evaluation  considers  inter-relationships  between  the  metrics.  There  are  some  issues  with 
this  approach.  First,  a  requirement  of  (n+1)  dimensional  space  is  mentioned  for  (n) 
number  of  different  metrics  as  depicted  in  Figure  1 1.  For  each  pair  of  metrics,  there 
would  be  9  different  possibilities,  as  depicted  in  Figure  12  as  dead,  penalty  and  reward 
zones.  However,  in  this  model,  only  two  metrics’  axes  were  presented  with  the  reward 
axis  in  order  to  define  the  zones  within.  So  basically,  as  the  authors  point  out,  using  the 

15  The  word  Figure  of  Merit  (FOM)  is  used  instead  of  metrics,  the  super-indices  L  and  U  denote  the  lower  and  the  upper  thresholds 
(Sols  et  al.,  2007). 
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five  key  parameters  as  DAU  introduced:  availability,  reliability,  logistics  foot  print, 
logistics  response  time  and  cost  per  unit  usage,  would  result  in  a  6-dimensional  space. 
This  implies  90  different  zones16  with  related  compensation,  penalty  and  reward 
amounts.  As  the  number  of  defined  metrics  increases  in  a  PBL  contract,  the  number  of 
zones  would  disproportionally  increase.  In  addition  to  being  hard  to  implement,  the 
Government  should  also  consider  the  risk  of  still  paying  some  amount  to  the  contractor 
because  of  the  existence  of  some  high  reward  zones.  A  similar  approach  is  used  by  some 
academics  to  suggest  possible  improvements  to  the  maintenance  contracting  process 
currently  utilized  by  the  South  Carolina  Department  of  Transportation  (SCDOT)  (Bell 
and  Berkland,  2007).  According  to  the  defined  8  key  parameters  and  44  derived  sub¬ 
parameters,  they  defined  a  weighted  evaluation  process  with  a  partial  payment  method 
that  includes  incentives.  Although  the  parameters  were  defined  in  a  systematic  way,  the 
evaluation  process  might  not  be  appropriate.  For  example,  SCDOT  accepted  to  pay  some 
amount  of  money  to  the  contractor  because  of  satisfying  some  parameters  at  certain 
levels;  however,  the  assessed  values  of  the  other  parameters  are  evaluated  as 
unacceptable  and  penalties  assessed.  (See  Table  3) 


6For  5  parameters;  first  we  need  to  select  2  from  5  and  then  we  need  to  define  9  zones  for  each  pair.  So  the  total  number  of  the  zones 
would  be: 


C 


x  9  =  90 
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Table  3.  Evaluation  Process  of  SCDOT 

(Gathered  from  Bell  and  Berkland,  2007) 


EXAMPLE  OF  CONTRACTOR  PAYMENT 
$  100.000.00  MONTHLY  UNIT  BID  AMOUNT 


Element  Value  equals  Element  Weight  X  Unit  Price 
Bid  (Monthly  Bid  Amount) 

Ex.  Asphalt  Pavement  Repair 
Element  Weight  =  09 
Unit  Price  Bid  =  $100,000.00 
ElementValue  =  S9.000.00 


if  the  contractor  performance  rating  meets  the  performance  target  then  the 
contractor  receives  total  payment  for  the  element  value.  If  the 
performance  rating  is  1  to  5  points  below  the  performance  target  then  the 
contract  payment  for  that  element  is  75%  of  the  element  value.  If  the 
performance  rating  is  6  to  10  points  below  the  performance  target  then  the 
contract  payment  for  that  element  is  50%  of  the  element  value.  If  the 
performance  rating  is  over  10  points  below  the  performance  target  then 
the  contractor  does  not  receive  payment  for  the  element. 


ELEMENT 

nm 

ELEMENT 

RATING- — i^LUE 
ELEMENT  BELOW — ► 

ELEMENT 

ELEMENT 

WEIGHT 

TARGET 

POINTS 

RATING 

POINTS 

TARGET 

VALUE 

PAYMENT 

Bridge  Decks 

0.0225 

80 

1.8 

59.7 

1.34325 

-20.3  S 

2,250  00 

S 

</) 

Superstructure 

0.045 

80 

3.6 

58.21 

2.61945 

-21.79  S 

4.500  00 

S 

111 

0 

Q 

Substructure 

0.045 

80 

3.6 

74.63 

3.35835 

-5.37  S 

4.500  00 

S 

2,250.00 

Pipes  &  Culverts 

0.0225 

90 

2.025 

47.19 

1.061775 

-42.81  S 

2,250.00 

S 

££ 

Retaining  Walls 

0.0075 

90 

0.675 

99.87 

0.675 

0  S 

750  00 

S 

750.00 

CD 

Channel  &  Slope  Protection 

0.0075 

90 

0.675 

62.69 

0.470175 

-27.31  S 

750.00 

s 

0.15 

on 

- TT575 

9.538 

Asphalt  Pavement  Repair 

0.09 

4  96 

8.55 

87.69 

7  8921 

-7.31  S 

9,000.00 

S- 

4,500.00 

2 

Concrete  Pavement  Repair 

0.09 

/  95 

8.55 

T  88.89 

8.0001 

/ 

-6.11  S 

9.000  QD-^ — 5 

4,500.00 

£ 

Pvm't  Shoulder  Condition 

002 

/  90 

1.8 

/  98.65 

1.8 

/ 

0  S 

jLOQOOO 

S 

2,000.00 

02 

/  94.5 

18.9 

17.6925 

Low  Shoulder 

0.04 

/  85 

3.4 

91.28 

T.4 

0  s 

4,000  00 

S 

4,000.00 

High  Shoulder 

0.03 

The  Element  Target  for  Asphalt  Pavement  Repair  is 

3,000.00 

S 

3,000.00 

S 5 

Latera  Ditches 

0.03 

95,  the  Contractor  Rating  is  87.69  Therefore,  the 

0  s 

3,000.00 

s 

3,000.00 

0.1 

Crossline  Pipe  (Blocked) 

0.03 

of  the  element  value  of  $9,000.00,  which  is 

0  s 

3,000.00 

s 

3,000.00 

UJ 

Crossline  Pipe  (Damaged) 

0.02 

$4,500.00 

0  $ 

2.000.00 

s 

2,000.00 

3 

Curb  &  Gutter  (Blocked) 

0.01 

0  s 

1.000  00 

s 

2 

Curb  &  Gutter  (Damaged) 

001 

85 

0.85 

99.94 

0.85 

0  s 

1.000  00 

5 

1,000.00 

O' 

Drop  Inlets.  CB's,  etc  (Blocked) 

0.02 

85 

1.7 

89.68 

1.7 

0  s 

2,000  00 

S 

2,000.00 

Q 

Drop  inlets,  CB's,  etc  (Damaged) 

0.01 

85 

0.85 

96.61 

0.85 

0  s 

1,000.00 

S 

1,000.00 

0.1 

85 

85 

8.5| 

UJ 

Guardrail/Cablerail 

009 

95 

8.55 

98.81 

8.55 

0  s 

9.000.00 

S 

9,000.00 

OADSID 

APPURT 

Concrete  Median  Barrier 

0.015 

95 

1.425 

99.92 

1.425 

0  s 

1,500.00 

S 

1,500.00 

Noee  Walls 

0.015 

95 

1.425 

80.84 

1.2126 

-14.16  S 

1.500.00 

s 

Impact  Attenuators 

0.03 

95 

29 

84.21 

2.5263 

-10.79  S 

3,000.00 

s 

cr 

0.15 

1371331 

Mowing 

0.03 

90 

2.7 

22.86 

0.6858 

-67.14  S 

3.000.00 

s 

Brush  &  Tree  Control 

0.0225 

85 

1.9125 

70.64 

1.5894 

-14.36  S 

2,250.00 

s 

0 

Turf  Condition 

0.015 

80 

1.2 

82.32 

1.2 

0  S 

1.500  00 

s 

1,500.00 

(/) 

Uncontrolled  Growth 

0.015 

70 

1.05 

76.22 

1.05 

0  S 

1,500  00 

s 

1,500.00 

$ 

Litter  &  Debris  Control 

0.0375 

90 

3.375 

5.71 

0.214125 

-84.29  S 

3,750  00 

s 

0 

Landscape  Beds 

0.0225 

70 

1.575 

63.64 

1.4319 

-6.36  S 

2,250.00 

s 

1,125.00 

CL 

Slope 

0.0075 

95 

0.7125 

99.26 

0.7125 

0  S 

750.00 

s 

750.00 

0.15 

84.75 

12.525 

6  883725 

Pavement  Markings 

0.03 

85 

2.55 

97.28 

2.55 

0  S 

3.000.00 

s 

3,000.00 

Pavement  Markers 

0.015 

85 

1.275 

97.31 

1.275 

0  S 

1,500.00 

s 

1,500.00 

1- 

Ground  Signs 

0.0195 

92 

1.794 

85.84 

1.67388 

-6  .16  S 

1.950.00 

s 

975.00 

0 

Overhead  Signs 

0.0375 

92 

3.45 

91.75 

3.440625 

-0.25  S 

3,750.00 

s 

2,812.50 

u. 

Roadway  lighting 

003 

90 

2.7 

47.19 

1.4157 

-42.81  S 

3.000  00 

s 

u. 

Words  %  Symbols 

0.0075 

85 

0.6375 

69.19 

0.518925 

-15.81  S 

750  00 

s 

§ 

Sign  Lighting 

0.0075 

90 

0.675 

41.45 

0.310875 

-48.55  S 

750  00 

s 

Delineators 

0.003 

90 

0.27 

89.31 

0.26793 

-0.69  S 

300.00 

s 

225.00 

0.1 5|  89.01 

U  453935 

2007  OVERALL  TARGET 

88.40 

76.27076 

OVERALL  CONTRACTOR 

AVAILABLE  MONTHLY 

PERFORMANCE 

76.27 

PAYMENT 

$100,000.00 

VALUE  BELOW 
PERFORMANCE 

TARGET 

12.13 

PERFORMANCE  PAYMENT 

$57,887.50 

Building  the  trust  and  partnerships  between  parties  is  an  important  aspect  of  PBL; 
however,  the  Government  side  must  consider  the  necessary  precautions  in  order  to 
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guarantee  the  desired  performance  outcomes.  One  of  the  most  effective  tools  for  the 
Government  is  using  incentives  and  disincentives. 

Sols  et  al.  (2008)  offered  an  n-dimensional  compensated  effectiveness  metric 
approach  with  a  reward  scheme  model,  which  proposes  to  calculate  a  single  number  as 
the  “compensated  system  effectiveness  metric”  and  build  a  reward  scheme  accordingly 
(Sols  et  al.  2008).  Defining  a  model  in  which  a  calculated  single  metric  for  the  evaluation 
process  proposed  can  be  useful.  Some  issues  should  be  considered.  The  metrics 
mentioned  are  composite  and  multiple  tiered  with  weights  assigned  at  each  tier  to  sub¬ 
components  of  the  effectiveness  measure.  Normalization  for  the  metrics  to  fit  in  [0,1] 
interval  pennits  aggregation.  The  responsibility  to  define  and  calculate  the  metrics  is 
given  to  the  customer;  this  may  include  some  degree  of  risk  because  of  the  subjectivity. 

In  addition,  compensating  between  pairs  of  metrics  is  complicated  when  some  are  over¬ 
performing  and  some  are  under-performing  especially  if  the  number  of  defined  metrics 
increases.  For  example,  a  real  life  example  of  an  SCDOT  contract  which  was  described 
by  Bell  and  Berkland,  (2007),  includes  44  sub-metrics;  therefore  SCDOT  could  consider 

1  7 

946  alternatives  before  implementing  the  compensation  between  metrics. 

Finally,  the  historical  or  the  predicted  value  of  the  calculated  system  effectiveness 
metric  indicated  the  process  for  defining  the  baseline,  minimum  acceptable  and  target 
values.  Both  historical  and  predicted  values  may  be  either  very  low  or  infeasible.  Instead, 
the  upper  and  the  lower  limits  of  the  values  should  be  defined  by  the  team  of  operational 
units,  according  to  their  needs  for  fulfilling  the  mission.  Then  the  contractors  should 
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search  the  feasibility  of  satisfying  those  needs  with  a  budget  constraint  via  their  expertise 
and  innovative  approaches. 

The  Core  Objectives  of  an  Evaluation  Model 

The  reviewed  literature  revealed  that  some  models  were  introduced  for  the 
evaluation  processes  in  PBL  and  some  of  the  models  are  mentioned  in  the  current  study. 
According  to  the  cumulated  knowledge,  the  objectives  of  a  model  which  would  be  used 
in  evaluation  processes  may  be  proposed  as: 

1)  Simplicity  :  The  model  must  be  easy-to-understand  and  apply  by  the  decision  makers. 

2)  Reasonability:  The  assumptions  and  the  results  of  a  model  must  make  sense  to  the 
decision  maker. 

3)  Elasticity  :  A  model  should  be  dynamic  and  can  be  applied  in  many  kinds  of  PBL 
contracts. 

4)  Precision  :  The  results  should  be  precise  to  guide  the  decision  maker. 

5)  Capability  :  A  model  should  be  capable  of  processing  any  number  of  metrics  within 
many  time  periods. 

6)  Integrality  :  A  model  should  be  capable  of  integrating  all  metrics  and  calculate  a  sole 
metric  as  the  overall  perfonnance  measure. 

7)  Objectivity  :  The  data  processing  and  the  outcomes  of  a  model  should  be  the 
same,  no  matter  who  performs  them.  Therefore,  the  model  should  be  quantitative  and 
closed  to  subjective  comments  as  much  as  possible. 
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Conclusion 


First  of  all,  PBL  was  born  from  the  necessities  of  the  new  era  of  war,  just  after  the 
9/11  terrorist  attacks  with  the  implementation  of  global  war  on  terrorism  (GWOT).  It 
represents  the  preferred  approach  for  all  procurements  by  DoD  and  can  be  defined  as:  the 
transformation  from  acquiring  materials  to  acquiring  performance  outcomes.  Although 
relying  on  the  knowledge  and  the  innovation  of  the  contractor  for  their  product  and/or 
service  may  be  reasonable,  the  benefits  of  PBL  have  not  been  experienced  yet,  because  of 
the  ineffective  implementation  for  this  long-tenn  support  strategy  (US  GAO,  2005). 

Second,  the  reviewed  literature  pointed  out  three  main  problem  areas  of  PBL  as: 
The  Ethical  Issues  of  Reliance  of  Contractors  in  the  Battlefields,  Issues  About  Contract 
Length,  and  Measurement  Issues  of  the  PBL  Systems.  Although  some  studies  have  been 
done  in  order  to  fill  the  gaps  in  these  areas,  there  are  still  some  issues  especially  for  the 
evaluation  processes  and  the  proposed  models.  Thus,  the  current  research  attempts  to 
contribute  to  the  literature  by  searching  for  a  systematic  approach  for  evaluation  of  the 
defined  metrics  via  a  theoretical  integrated  model  to  monitor  the  system  and  to  make 
healthy  decisions  about  it. 

Finally,  the  core  objectives  of  a  model  were  suggested  as:  reasonability,  elasticity, 
precision,  capability,  integrality  and  objectivity.  Achieving  these  objectives  is  an  aimed 
of  the  current  study. 
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III.  Methodology 


Introduction 

First  of  all,  the  PBL  concept  was  studied  according  to  the  published  literature  and 
experts’  opinions.  Second,  the  gaps  of  the  PBL  literature  were  searched  and  defined  in 
order  to  identify  a  contribution  in  one  of  those  directions.  This  research  suggests  a 
systematic  approach  via  an  integrated  model  to  combine  multiple  criteria  when  assessing 
the  success  of  a  PBL  contract. 

A  4-Step  Process  Evaluation  Model  (the  4SPEM)  is  proposed  in  this  thesis.  First 
the  current  evaluation  model  (CEM)  is  built.  The  CEM  is  used  to  check  the  achieved 
results  of  individual  metrics  according  to  whether  or  not  they  are  satisfying  the 
contractual  requirements.  Second,  a  mathematical  model  (MM)  is  built.  The  MM  is  used 
to  calculate  the  overall  performance  metric  (OPM)  for  each  term.  Briefly,  the  OPM  is  a 
single,  integrated  number  which  represents  the  overall  perfonnance  of  the  system  within 
a  defined  time  period  (term).  Third,  the  integrated  model  (IM)  is  built.  The  IM  is  used  to 
put  together  the  findings  of  the  CEM  and  MM  and  the  findings  of  the  IM  are  analyzed 
and  interpreted. 

The  defined  objectives  of  the  4SPEM  are  achieving:  reasonability,  elasticity, 
precision,  capability,  integrality  and  objectivity. The  detailed  methodology  is  depicted  in 
Figure  13. 
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Figure  13.  The  Methodology  Algorithm  of  the  Thesis 


4-Step  Process  Evaluation  Model  (the  4SPEM) 

The  4SPEM  is  designed  to  suggest  an  additional  insight  to  decision  makers  about 
the  PBL  system  that  is  governed  by  them.  The  milestones  of  the  4SPEM  are  defined  as: 
Building  the  Current  Evaluation  Model  (CEM),  Building  the  Mathematical  Model  (MM), 
Building  the  Integrated  Model  (IM)  and  Analysis  and  Interpretation  of  the  Findings. 
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Step  1:  Building  the  Current  Evaluation  Model  (CEM) 

The  CEM  is  built  according  to  the  current  evaluation  methods  which  are  used  by 
the  acquisition  professionals.  Although  the  literature  reviewed  so  far  suggests  that  there’s 
no  template  for  evaluation  of  the  parameters,  the  evaluation  in  the  CEM  is  by  a  2-step 
process,  as  described  below: 

1)  Collecting  data  of  assessed  outcome  values  of  defined  metrics  by  time  period. 

2)  Comparison  of  each  calculated  metric  assessed  value  with  the  defined  target  value  of 
that  metric  in  the  contract. 

In  addition  to  this  2-step  process,  some  academics  mention  applying  weighting  of 
metrics  for  the  evaluation  process  (Canada  and  Sullivan,  1989).  Although  weights  can  be 
applied  within  the  CEM,  it  would  decrease  the  level  of  objectivity  of  the  model.  In 
addition  to  that,  favoring  one  metric  over  another  may  lead  the  contractor  in  the  wrong 
direction.  The  viewpoint  taken  for  the  current  research  is  that  all  of  the  metrics  should  be 
perceived  as:  the  defined  needs  of  the  war  fighter.  Thus,  none  of  them  should  be  treated 
differently  than  the  others. 

In  conclusion,  the  CEM  will  be  used  for  the  first  step  to  check  whether  or  not  the 
minimum  contractual  requirements  are  satisfied  by  period/tenn. 

Step  2:  Building  the  Mathematical  Model  (MM) 

First  of  all,  defining  a  desired  metric  value  within  a  time  frame  is  the  same  as 
defining  an  outcome  of  a  time  function.  By  definition:  “a  function  is  a  rule  that  assigns  to 
each  input  number  exactly  one  output  number.”  (Haeussler  et  ah,  2007) 
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Secondly,  the  importance  of  PBL  is  demanding  many  different  perfonnance 
outcomes  simultaneously  at  the  end  of  certain  time  periods  continuously.  It  can  be 
illustrated  with  an  explosion  of  a  bomb  example.  Let’s  assume  that  the  bomb  was 
triggered  at  time  zero  and  as  the  time  passes  the  blast  expands  in  all  directions 
simultaneously  and  the  effect  (the  overall  perfonnance  outcome)  of  the  bomb  is 
explained  by  the  multidimensional  expansion  of  the  blast  in  the  space. 

Every  metric  in  the  PBL  is  a  defined  dimension  in  the  space  and  they  will  expand 
in  different  directions  following  the  triggering  of  the  system.  The  periods  of  time  at 
which  we  calculate  the  assessed  values  of  the  metrics  are  the  snap  shots  of  this 
continuous  process.  So  in  addition  to  calculating  the  assessed  values  of  the  metrics 
individually,  we  can  also  calculate  the  overall  performance  outcome  (the  expansion)  of 
the  system. 

The  MM  proposes  to  calculate  and  analyze  a  sole  number,  the  overall 
performance  metric  (OPM),  for  both  the  overall  contractual  expectation  and  the  assessed 
overall  perfonnance  of  the  system.  These  calculated  numbers  will  be  used  to  evaluate 
whether  or  not  the  overall  expectation  from  the  relevant  metrics  at  defined  times  was 
addressed.  Briefly,  by  defining  the  requirements  for  satisfying  each  parameter  within  a 
time  frame,  a  multi-dimensional  shape  in  the  space  is  defined  upfront  in  the  contract.  For 
example,  if  three  parameters  were  defined  in  a  PBL  contract,  then  a  four-  dimensional 
volume  is  defined  and  the  volume  of  the  shape  is  the  OPM  of  contract.  The  fourth 
dimension  is  time. 

The  MM  model  depends  on  a  4-step  process.  The  first  step  is  building  the 
customized  integration  fonnula  (GIF)  of  the  relevant  metrics  defined  in  the  contract.  The 
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important  point  of  building  the  integration  formula  is  defining  the  effect  of  each  metric  to 
the  system.  If  it  has  a  positive  effect,  then  the  time  function  representing  that  metric  will 
be  used  in  the  numerator  for  its  part  in  the  fonnulation;  otherwise  it  will  be  used  in  the 
denominator.  For  example,  Ao  (t)  and  R  (t)  are  desired  to  be  as  much  as  possible  (positive 
effect),  but  LRT  (t)  is  desired  to  be  as  low  as  possible  (negative  effect).  So  Ao  (t)  and  R 
(t)  will  be  used  in  the  numerator,  but  LRT  (t)  will  be  used  in  denominator.  Equation  (1) 
illustrates  an  example  formulation  of  a  customized  integration  fonnula  to  calculate  the 
overall  performance  metric. 


OPM  = 


n  n  n  + 

J  A  ( t)d  (t)  xfpa  ( t)d  (?)  x  J  —— —  dt 

g  g  g  ^  ' 


where 


(1) 


A0  (t),  R(,  (t)  and  LRT  (t)  are  relevant  time  functions  of  the  defined  metrics, 
g  is  the  beginning  of  the  term  and  h  is  the  end  of  the  term. 


The  second  step  is  revealing  the  time  functions  of  the  parameters  such  as:  Ao  (t) 

,  R  (t)  and  LFP  (t).  Equations  (2)  to  (4)  are  used  to  calculate  the  coefficients  of  the 
independent  variables.  Briefly,  Equation  (2)  is  the  general  exhibition  of  the  time 
functions  of  each  metric,  Equation  (3)  is  the  matrix  fonn  of  the  time  functions  and 
Equation  (4)  is  the  calculation  of  the  coefficients  of  the  relevant  time  functions. 

/(/)  =  ^  (2) 

i= 1 

where 

n  =  the  number  of  terms  that  performance  data  is  evaluated  for  individual  metrics 
x;  =  the  coefficients  of  independent  variables  in  the  function 
t  =  the  independent  variables  in  the  function 
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yk=  the  dependent  variable  in  the  function 


Since  we  know  the  values  of  t,  n  and  yk ,  we  only  need  to  calculate  x,  values  in 
order  to  reveal  the  time  functions  of  the  relevant  metrics.  Hence,  we  will  transfonn  the 
general  exhibition  of  time  functions  to  matrix  fonn  as  depicted  in  Equation  (3). 


f  (t)  =  Ax  B  =  C 


where 


A  =  matrix  form  of  independent  variables 
B  =  matrix  form  of  coefficients 
C  =  matrix  form  of  dependent  variables 


hence 


rr> 


A  = 


vr 


nJ 


,  B  = 

V 

,  c  = 

V 

nxn 

v*J 

nx  1 

(3) 


Then  we  will  use  the  mathematical  approach  depicted  in  Equation  (4)  to  calculate 
the  values  of  coefficients  of  the  independent  variables. 

AxAlxB  =  CxA (4) 

where 

A'1  =  inverse  of  matrix  A 

Third,  Equations  (2),  (3),  (4)  and  (1)  are  used  iteratively  to  calculate  the  volume 
of  the  PBL  shape  for  the  relevant  terms,  in  other  words,  the  OPMs  for  the  relevant  terms 
of  the  PBL  contract. 

We  defined  two  different  ways  of  finding  the  time  functions  of  the  relevant 
metrics  via  Equations  (2),  (3),  (4)  and  applying  them  to  Equation  (1)  to  calculate  OPMs. 

The  first  way  may  be  calculating  continuous  time  functions  for  each  metric  from 
point  zero  at  the  time  scale  to  the  actual  point  where  the  latest  tenn  is  via  Equations  (2), 
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(3)  and  (4).  The  time  functions  are  quadratic  after  the  first  tenn  in  this  case  and  then 
Equation  (1)  (GIF)  is  applied  via  using  the  latest  tenn’s  and  the  previous  term’s  values  of 
time  on  the  time  scale  as  boundaries  of  integrals  to  calculate  the  OPM  of  the  relevant 
tenn.  The  sum  of  all  tenns’  OPMs  will  give  us  the  overall  OPM. 

The  second  way  may  be  calculating  discrete  time  functions  for  each  metric  from 
the  previous  term  to  the  latest  term  via  Equations  (2),  (3)  and  (4).  The  time  functions  will 
be  linear  in  this  case  for  all  terms.  Thus,  Equation  (1)  (GIF)  are  applied  via  using  only  0 
(zero)  and  l(one)  as  boundaries  of  integrals  to  calculate  OPMs. 

Finally,  we  will  try  to  evaluate  both  approaches  according  to  the  analysis  of  the 
data.  After  that  we  will  analyze  each  term  via  comparing  the  values  of  contractually- 
demanded  OPMs  with  assessed  values  of  OPMs  and  then  the  findings  will  be  evaluated. 

Step  3:  Building  the  Integrated  Model  (IM) 

The  IM  is  built  via  combining  and  organizing  the  findings  of  the  GEM  and  the 
MM.  A  table  fonnat  is  used  to  exhibit  the  IM  which  reveals  the  current  situation  of  the 
system  and  the  trend  of  the  system.  By  this  way,  the  decision  makers  would  gain  more 
insights  about  the  system  for  which  they  are  responsible.  Additionally,  the  IM  would  be 
used  as  a  mediator  tool  between  the  contractor  and  the  Government  for  both  negotiations 
and  working  on  the  solutions  of  the  conflicts.  Thus,  it  may  be  useful  to  build  trust 
between  the  partners. 

Step  4:  Analysis  and  Interpretation  of  the  Findings 

By  the  highly-elastic  nature  of  the  PBL,  every  IM  is  unique.  However,  concluding 
all  of  the  findings  gathered  from  either  the  GEM,  or  the  MM,  to  see  the  big  picture  before 
the  decision-making  is  the  general  idea  of  the  IM. 
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Thus,  the  IM  will  be  analyzed  in  order  to  understand  both  the  overall  performance 
and  the  general  trend  of  the  system.  Finally,  the  conclusions  depending  on  the  evaluation 
of  results  will  be  interpreted. 

Data  Sources  /  Format 

First  of  all,  one  of  the  aims  of  the  current  study  is  to  test  the  applicability  and  the 
usefulness  of  the  4SPEM.  Thus,  PBL  data  from  published  sources  was  used.  By  doing  so; 
examination  of  the  results  would  benefit  from  the  insights  gathered  via  the  studies  of  the 
previous  researchers  about  the  PBL  contracts  that  they  worked  on.  Additionally,  the 
model  in  the  current  study  may  be  evaluated  with  real  life  data. 

Data  Analysis 

Two  common  technological  tools:  Microsoft  Excel  and  Mathcad  14  are  used  to 
process  the  data,  and  perform  the  relevant  analyses.  However,  the  approach  is  not  limited 
to  these  two  applications.  Many  kinds  of  mathematical  tools  including  scientific 
calculators  may  be  used. 

The  Example  Study 

First  of  all  in  this  example  study,  the  usage  of  numbers  and  the  mathematical 
equations  are  used  as  little  as  possible  for  the  sake  of  simplicity.  The  aim  of  this  example 
is  to  provide  a  better  understanding  of  this  thesis. 
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Let’s  assume  that  the  contract  is  agreed  between  the  two  sides  and  at  the  end  of 
each  year  the  collected  parameters  are  evaluated  according  to  the  CEM  and  all  of  them 
were  satisfied  according  to  the  defined  contract  requirements.  Can  we  assume  that  the 
contractor  is  successful  and  we  would  continue  with  that  contractor?  How  about, 
although  some  parameters  are  in  an  upward  trend,  the  others  are  declining  or  may  be  no 
different  at  all?  The  answers  of  these  questions  and  the  expected  comparison  between 
CEM  and  IM  are  depicted  in  Table  4.  18 


18  Contract  metrics  are  satisfied  for  all  terms.  And  they  are  compared  with  the  contract  and  previous  term;  and  increase  is  defined  as 

“  /  ’’decrease  is  defined  as  “  and  no  difference  is  defined  as  “  and  it’s  assumed  that  the  overall  performance  metric  is 
calculated  as  1 .5  from  the  contract  according  to  MM. 
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Table  4.  Theoretical  Comparison  of  CEM  with  IM 


1“  YEAR  with  CEM 

1“  YEAR  with  IM  (individual  metrics  &  overall  performance  metric) 

Availability 

t 

Availability 

t 

Ov  erall  Performance  Metnc  (OPM) 

Reliability 

t 

Reliability 

t 

1.57 

Cost 

i 

Cost 

1 1 

Logistics  footprint 

Logistics  footprint 

— 

Logistics  response  time 

i 

Logistics  response  time 

i 

2*  YEAR  with  CEM 

2“*  YEAR  with  IM 

Availability 

— 

Availability 

— 

OPM 

Reliability 

t 

Reliability 

1 

1.59 

Cost 

t 

Cost 

t 

Logistics  footprint 

t 

Logistics  footprint 

t 

Logistics  response  time 

t 

Logistics  response  time 

t 

3  rd  YEAR  with  CEM 

3*  YEAR  with  IM 

Availability 

t 

Availability 

t 

OPM 

Reliability 

i 

Reliability 

1 1 

Cost 

t 

Cost 

t 

1.6 

Logistics  footprint 

— 

Logistics  footprint 

— 

Logistics  response  time 

— 

Logistics  response  time 

— 

END  OF  3'd  YEAR 

END  OF  3ri  YEAR 

How’sour  system  going?  (hardto 

How’s  our  system  going?(easy  to  measure  with  additional  help  of  my  model) 

measure) 

1  5<1J7<L59<1 .6 

How's  our  contractor  doing'’  fit 
depends) 

How-’sour  contra  ctordoingi’(he's  doing  fine  accordingto  my  model) 

Will  we  continue  with  him? 

1. 5<L57<L59<1 .6 

(subjective) 

Will  w-e  continue  withhim?(his  performance  has  anupward trend,  so  yes) 

L5<L57<L59<L6 
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Conclusion 


In  this  chapter,  the  methodology  of  the  thesis  is  briefly  explained.  First,  the  map 
of  the  study  is  described  via  the  Methodology  Algorithm  in  Figure  13.  A  holistic 
approach  is  adopted  and  the  study  is  concentrated  on  the  post-PBL  period.  The  aim  of  the 
current  study  is  to  contribute  to  the  area  of  evaluation  processes  and  propose  a  model 
which  should  include:  reasonability,  elasticity,  precision,  capability,  integrality  and 
objectivity. 

Second,  a  4-Step  Process  Evaluation  Model  (the  4SPEM)  is  defined.  It  starts  with 
building  the  Current  Evaluation  Model  (CEM),  which  has  a  2-step  sub  process.  It 
continues  with  building  the  Mathematical  Model  (MM).  A  4-step  sub  process  is  defined 
to  realize  the  MM  with  the  underlying  theories.  However,  we  defined  two  different 
approaches  in  building  the  MM  and  we  will  use  one  of  the  two  approaches  after  the 
evaluation  process.  Next,  building  the  Integrated  Model  (IM)  is  defined  and  the 
interpretation  of  the  findings  explained.  Following  that  the  expected  data  sources  are 
explained  with  the  format  of  the  data  and  then  the  planned  approach  of  the  data  analysis 
is  defined. 

Finally,  a  simple  example  of  the  thesis  is  illustrated  in  Table  4.  The  aim  of  the 
example  is  to  provide  a  better  understanding  of  the  current  study;  therefore,  the  usage  of 
numbers  and  mathematical  formulas  were  limited. 
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IV.  Analysis 


Introduction 

First  of  all,  this  research  chose  two  real  life  PBL  cases  from  published  sources  in 
order  to  apply  the  4SPEM.  Briefly,  Case  1  is  related  to  the  Navy  A-RCI19  and  Virginia 
S/CC/A  Program.  The  research  of  Gansler  and  Lucyshyn  (2006)  is  used  for  both 
explaining  and  gathering  the  data  of  performance  metrics  of  the  program.  Case  2  is 
related  to  the  C-17  Globemaster  Sustainment  Program  and  the  research  of  Mahon  (2007), 
is  used  as  the  data  source  for  that  case. 

Secondly,  the  4SPEM"  is  applied  to  both  cases  and  the  applicability  and  the 
reliability  of  the  model  is  checked.  Microsoft  Excel  and  Mathcad  14  are  used  as  technical 
tools;  however,  other  tools  may  be  used  according  to  the  familiarity  of  the  researcher. 

The  analyses  and  interpretation  of  the  findings  are  done  according  to  the  4SPEM 
at  the  end  of  each  case.  The  primary  concern  is  investigating  whether  or  not  the  approach 
in  the  current  study  may  reveal  additional  insights  about  the  relevant  PBL  contracts. 


19  “The  A-RCI  is  a  four-phase  program  for  transforming  submarine  sonar  systems  (AN/BSY-1,  AN/BQQ-5  and  AN/BQQ-6)  from 
legacy  systems  to  a  more  capable  and  flexible  COTS/Open  System  Architecture  (OSA)  and  is  designated  AN/BQQ-10"  (Gansler  and 
Lucyshyn,  2006,  pp.  28) 

20  Virginia  Class  sonar, combat  control  and  architecture  (S/CC/A) 

21  the  4SPEM  includes;  Building  the  CEM,  Building  the  MM,  Building  the  IM  , Analyzing  and  Implementing  the  Findings 
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Case  1 


The  Navy  A-RCI22  and  Virginia  S/CC/A23  Program; 

This  PBL  program  is  responsible  for  managing  the  entire  Acoustic  Rapid  Commercial 
off  the  Shelf  Insertion  (A-RCI),  and  Sonar  Combat  Control  and  Architecture 
(S/CC/A)  supply  chain  process,  from  processing  the  requisition  through  the 
transportation  to  the  fleet  or  the  point  of  debarkation  (for  overseas  delivery).  The 
inventory  is  stored  and  repaired  at  the  contractor’s  Manassas  facility  (Gansler,  2006, 

P  31). 

1 .  The  desired  performance  outcomes  are  mentioned  in  the  PBL  contract  as: 

a.  “Supply  Material  Availability  (SMA):  The  contractual  requirement  for  SMA  is 
mentioned  as  85  percent”  (Gansler,  2006,  p  31  ). 

b.  “Average  Customer  Wait  Time  (ACWT):  ACWT  requirement  for  issue  priority 
group  (IPG)  I  items  for  continental  U.S.  (CONUS)  is  2  days. 

ACWT  for  IPG  I  items  for  outside  the  continental  U.S.  (OCONUS)  is  3  days. 
ACWT  requirements  for  IPG  II  items  for  CONUS  and  OCONUS  are  mentioned 
as  within  the  required  delivery  date”  (Gansler,  2006,  p  3 1). 

The  contractual  requirements  are  depicted  in  Table  5. 


Table  5.  Contractual  Requirements  for  Case  1 

(Gathered  from  Gansler  and  Lucyshyn,  2006) 


Monthly 

S-04 

0-04 

N-04 

D-04 

J-05 

F-05 

M-05 

A-05 

Terms 

1 

2 

3 

4 

5 

6 

7 

8 

SMA 

85% 

85% 

85% 

85% 

85% 

85% 

85% 

85% 

ACWT 

(CONUS) 

2 

days 

2  days 

2  days 

2  days 

2 

days 

2 

days 

2  days 

2  days 

ACWT 

(OCONUS) 

3 

days 

3  days 

3  days 

3  days 

3 

days 

3 

days 

3 

days 

3 

days 

22  The  A-RCI  is  a  four-phase  program  for  transforming  submarine  sonar  systems  (AN/BSY-1,  AN/BQQ-5  and  AN/BQQ-6)  from 
legacy  systems  to  a  more  capable  and  flexible  COTS/Open  System  Architecture  (OSA)  and  is  designated  AN/BQQ-10.  (Gansler  and 
Lucyshyn,  2006) 

23  Virginia  Class  sonar, combat  control  and  architecture  (S/CC/A) 
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2.  The  achieved  performance  of  the  related  metrics  for  the  first  eight  months  were  as 
depicted  in  Table  6; 


Table  6.  Achieved  Average  Performance  Results  for  Case  1 


(Gathered  from  Gans 


er,  2006) 


Monthly 

S-04 

0-04 

N-04 

D-04 

J-05 

F-05 

M-05 

A-05 

Periods 

1 

2 

3 

4 

5 

6 

7 

8 

SMA 

92.1% 

89.5% 

91.1% 

86.4% 

100.0% 

97.5% 

97.6% 

96.4% 

ACWT 

1.8 

2.0 

1.1 

2.3 

1.7 

1.6 

1.8 

1.9 

(CONUS) 

days 

days 

days 

days 

days 

days 

days 

days 

ACWT 

1.9 

2.1 

2.1 

2.5 

2.0 

2.5 

2.6 

2.5 

(OCONUS) 

days 

days 

days 

days 

days 

days 

days 

days 

According  to  the  data  gathered  from  Gansler  (2006),  the  evaluation  process  of  the 
Program  office  depends  on  defining  the  needs  in  the  contract,  reflecting  them  via 
different  metrics  and  calculating  each  term’s  assessed  values,  and  comparing  them 
individually  at  the  end.  However,  checking  the  assessed  values  with  the  contractual 
requirements  indicates  that  there  is  a  conflict  between  AWCT  (CONUS)  and  the  others  at 
the  forth  month  (D-04).  Case  1  is  analyzed  with  the  same  data  via  the  4SPEM. 

The  4SPEM  for  Case  1 

Step  1:  Building  the  Current  Evaluation  Model  (CEM). 

According  to  the  CEM  model,  there  are  conflicts  between  SMA,  ACWT 
(OCONUS)  and  ACWT  (CONUS)  at  the  fourth  period.  The  contractor  satisfied  the 
requirements  for  two  metrics;  however,  the  requirements  of  AWCT  (CONUS)  were  not 
satisfied.  (See  Table  7) 
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Table  7.  The  CEM  Model  of  Case  1 


Metrics 

SMA 

ACWT  (CONUS) 

ACWT  (OCONUS) 

Terms 

Bottom- 

line 

Actual- 

average 

Satisfied 

Bottom- 

line 

Actual- 

average 

Satisfied 

Bottom- 

line 

Actual- 

average 

Satisfied 

S-04 

1 

85% 

92.1% 

YES 

2  days 

1.8 

days 

YES 

3  days 

1.9 

days 

YES 

0-04 

2 

85% 

89.5% 

YES 

2  days 

2.0 

days 

YES 

3  days 

2.1 

days 

YES 

N-04 

3 

85% 

91.1% 

YES 

2  days 

1.1 

days 

YES 

3  days 

2.1 

days 

YES 

D-04 

4 

85% 

86.4% 

YES 

2  days 

2.3 

days 

NO 

3  days 

2.5 

days 

YES 

J-05 

5 

85% 

100.0% 

YES 

2  days 

1.7 

days 

YES 

3  days 

2.0 

days 

YES 

F-05 

6 

85% 

97.5% 

YES 

2  days 

16 

days 

YES 

3  days 

2.5 

days 

YES 

M-05 

7 

85% 

97.6% 

YES 

2  days 

1.8 

days 

YES 

3  days 

2.6 

days 

YES 

A-05 

8 

85% 

96.4% 

YES 

2  days 

1.9 

days 

YES 

3  days 

2.5 

days 

YES 

The  CEM  of  Case  1  reveals  that  there’s  a  conflict  at  the  fourth  month  (D-04), 
because  all  performance  metrics  satisfied  contractual  requirements  except 
ACWT  (CONUS).  In  addition,  all  of  the  metrics  have  waving  patterns.  In  conclusion,  the 
overall  perfonnance  trend  of  the  system  is  hard  to  reveal. 


57 


Step  2:  Building  the  Mathematical  Model  (MM). 

First  of  all,  ACWT  (CONUS)  and  ACWT  (OCONUS)  are  measuring  the  same 
behavior  which  is  the  Logistics  Response  Time  (LRT).  Second,  the  achieved  values  of 
ACWT  (OCONUS)  satisfied  the  contractual  requirements.  Thus,  the  MM  is  built  via 
using  two  metrics  which  are:  SMA  and  ACWT  (CONUS)  for  illustration. 

Additionally,  the  conflict  experienced  at  the  fourth  month  (D-04)  can  be  resolved 
via  proposing  the  OPM  which  would  integrate  the  effects  of  two  parameters  into  one 
metric. 

Building  the  Customized  Integration  Formula  (CIF)  for  Case  1. 

First  of  all,  the  effect  of  each  metric  to  the  system  should  be  understood  in  order 
to  build  the  CIF;  however,  it  is  not  necessary  to  calculate  the  correlations  of  the  metrics. 
Defining  an  individual  metric  as  bigger  is  better  (+  effect),  or  smaller  is  better  (-  effect), 
is  enough  to  build  the  formula.  (See  Table  8) 


Table  8 

The  Effects  of  the  metrics 

Metrics 

The  bigger  is 
better? 

The  effect  of  the 
metric 

SMA 

YES 

(+) 

ACWT  (CONUS) 

NO 

(-) 

The  CIF  is  built  according  to  defined  effects  of  the  metrics  for  Case  1  using  Equation  (1) 
and  it  is  depicted  in  Equation  (5). 
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OPM  =  f  SMA(t)jd(t)x  f - dt 

{  J  AWCT(t) 


where 


OPM 
SMA  (t) 
AWCT  (t) 

g 

h 


=  the  overall  performance  metric 

=  time  function  of  SMA  metric 

=  time  function  of  AWCT  metric 
=  the  beginning  of  the  term 
=  the  end  of  the  tenn 


(5) 


Finding  the  Time  Functions  of  the  Metrics. 

Two  approaches  are  defined  for  finding  time  functions.  In  the  first  approach  a 
continuous  quadratic  time  function  (CQTF)  is  built  throughout  the  time  scale  from  point 
zero  to  the  latest  data  point  for  each  tenn.  In  the  second  approach,  a  discrete  linear  time 
function  (DLTF)  is  built  from  the  previous  data  point  to  the  current  data  point  for  each 
period.  The  following  steps  guide  both  approaches. 

First  the  contractually-defined  time  functions  of  the  parameters  are  calculated  for 
each  term  of  contract.  Then  the  time  functions  of  the  achieved  perfonnance  results  are 
calculated  for  each  term  of  the  contract.  Equations  (2),  (3)  and  (4)  are  used  iteratively  to 
calculate  the  time  functions  of  each  metric  for  the  relevant  terms  via  the  Mathcad  14 
program  and  the  end  of  the  1st  term  (S-04)  is  accepted  as  the  origin  time  (time  zero)  in 
our  calculations.24 


24  Since  we  don’t  have  the  performance  data  of  the  previous  term  before  S=04,  we  started  from  S-04.  Therefore,  S-04 
=  t]=0,  0-04  =  t2=l,  ...,  A-05=  t8  =7 
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The  time  functions  are  unique  both  for  each  metric  and  for  each  term.  This 
reflects  reality,  because  in  the  real  life,  activities  that  take  place  to  satisfy  the  contractual 
agreement  are  not  stopped  before  the  evaluation.  Besides,  a  snapshot  picture  is  taken  and 
the  evaluation  is  made  while  the  system  continuously  keeps  moving  and  the  time 
functions  are  evolving  relative  to  the  data  cumulated. 

In  this  case,  first  the  continuous  quadratic  time  functions  (CQTF)  is  found  at  the 
end  of  each  term  for  each  metric  for  both  the  contract  and  the  assessed  values. 

1)  Building  Continuous  Quadratic  Time  Functions  (CQTF) 

a)  The  contractually-defined  CQTF’s  of  the  metrics  for  Case  1 

Both  metrics  are  defined  to  be  satisfied  as  constant  values  at  the  bottom-line  as 
seen  in  Table  7.  Thus,  they  are  represented  as  depicted  in  Equations  (6)  and  (7)  for  all 
terms  without  changing. 

SMAc(t)  =  S5t°  (6) 

where 

SMA(t)  =  time  function  of  SMA  metric 

AWCTc(t)  -  2t°  (7) 

where 

AWCT(t)  =  time  function  of  AWCT  metric 

b)  The  assessed  CQTF’s  of  the  Metrics  for  Case  1 

After  calculating  the  contractually-defined  time  functions  of  the  parameters,  the 
time  functions  of  the  assessed  values  of  the  parameters  are  calculated  according  to  the 
actual  annual  averages  of  the  parameters. 
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The  iterative  calculations  defined  in  Equations  (2),  (3)  and  (4)  are  made  via 
Matcad  14  to  reveal  the  time  functions  for  the  assessed  values  of  the  defined  metrics  of 


the  relevant  terms.  The  calculations  of  first  four  terms  are  depicted  in  Appendix  A. 

The  relevant  time  functions  of  the  tenns  are  depicted  in  Equation  (8),  through 
Equation  (21). 

SMAal(t)  =  -26?  +  92.1  (8) 

where 

SMAai  (t)  =  time  function  of  SMA  metric  for  the  1st  term 

AWCTaX{t)  =  Q2t  +  \.%  (9) 

where 

AWCTai  (t)  =  time  function  of  AWCT  metric  for  the  1st  tenn 

SMAa2(t)  =  -2.1r  -47? +  92.1  (10) 

where 

SMAa2  (t)  =  time  function  of  SMA  metric  for  the  2nd  tenn 

AWCTa2(t)  = -0.55t2  +  0.75?  +  1.8  (11) 

where 

AWCTa2  (t)  =  time  function  of  AWCT  metric  for  the  2nd  term 

SMAa3(t )  =  -1 ,75f3  +  7.35f2  -  8.2f  +  92.1  (12) 

where 

SMAa3  (t)  =  time  function  of  SMA  metric  for  the  3rd  tenn 
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(13) 


AWCTa3(t )  =  -0.533/3  -  2.15/2  +  1.817?  + 1.8 

where 

AWCTa3  (t)  =  time  function  of  AWCT  metric  for  the  3rd  term 

SMAa4(t )  =  1.462/4  -10.52/3  +23.43 It2  -1 6.975/  +  92.1  (14) 

where 

SMAa4  (t)  =  time  function  of  SMA  metric  for  the  4th  tenn 


AWCTa4(t)  =  -0.296/4  +  2.208/3  -5.404/2  +  3.592/ +  1.8  (15) 

where 

AWCTa4  (t)  =  time  function  of  AWCT  metric  for  the  4th  tenn 


SMAaS(t )  =  -0.784/5  +  9.304/4  -37.97/3  +  62.646/2 
-35.795/ +  92.1 

where 

SMAa5  (t)  =  time  function  of  SMA  metric  for  the  5th  tenn 


(16) 


A  WCTaS (; t )  =  0.11  It5  - 1 .404/' 4  +  6.1 87/3  - 10.946/2 
+  6.252/ +  1.8 

where 


(17) 


AWCTa5  (t)  =  time  function  of  AWCT  metric  for  the  5th  tenn 


SMAa6(t )  =  0.297/6  -  5.253/5  +  34.683/4  - 105.256/3  + 144.675/2 
-7 1.746/ +  92.1 

where 

SMAa6  (t)  =  time  function  of  SMA  metric  for  the  6th  tenn 


(18) 
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(19) 


AWCTa6(t )  =  -0.032*6  +  0.558*5  -4.ll2t4  +  13.366*3 
-19.698*2  +10.087*  +  1.8 

where 

AWCTa6  (t)  =  time  function  of  AWCT  metric  for  the  6th  tenn 


SMAal(t)  =  0.028?/ -0.3 174- 3.577  xKT3*35  + 12. 147*44 
-54.586*53  +88.9 14*62  -48.7834  +  92.1 

where 

SMAa7  (t)  =  time  function  of  SMA  metric  for  the  7th  tenn 


(20) 


A  WCTal(t)  =  -3. 179  x  10"34  +  0.00374  +  3-267  x  lO-4^5 
-1.5914  +  7.697*53  -13.459*62  -7.5184  +1.8 

where 


(21) 


AWCTa7  (t)  =  time  function  of  AWCT  metric  for  the  7th  tenn 


Calculation  of  OPMs  for  CQTF 

Since  we  calculated  all  the  necessary  time  functions  of  the  relevant  metrics  for 
each  tenn,  we  can  calculate  the  OPMs  for  each  term  both  for  the  contractual  and  the 
assessed  performance  values  using  Equation  (5). 

Table  9  organizes  the  calculated  OPMs  for  both  the  contract  and  the  assessed 
values.  The  mentioned  terms  in  the  table  refer  to  the  time  frame  between  two  consecutive 
tenns  of  Table  7.  For  example,  1st  tenn  of  Table  9  covers  the  time  frame  between  S-04 
and  0-04  of  Table  7.  AOPMs  refer  to  the  difference  between  OPM  ASSESSED  and  OPM 
CONTRACT  in  the  table. 
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Table  9.  The  Calculated  Values  of  OPMs  via  CQTF 


TERMS 

OPM  CONTRACT 

OPM  ASSESSED 

A  OPM’s 

1st 

42.5 

47.834 

5.334 

2nd 

42.5 

56.377 

13.877 

3rd 

42.5 

68.43 

25.93 

4th 

42.5 

4656 

4613.5 

5th 

42.5 

75.133 

32.633 

6th 

42.5 

22.992 

-19.508 

'yth 

28.863 

23.298 

-5.565 

All 

283.863 

4950.064 

4666.2 

Analysis  of  CQTF-included  MM 

The  MM  Model  with  CQTF  Case  1  reveals  that  ACWT  (CONUS)  is  not  satisfied 
for  the  contractual  requirements  according  to  D-04  (4th  month)  at  the  CEM.  The  OPM 
satisfied  the  contractual  requirements  according  to  the  contract  between  3ld  and  4th 
periods. 

However,  a  fluctuation  is  experienced  at  the  4th  period.  The  reflection  of  the 
previous  term’s  perfonnance  effects  might  be  considerable,  but  the  amount  of  fluctuation 
makes  it  almost  impossible.  In  addition  to  that,  although  the  contractor’s  performance  for 
both  metrics  was  much  higher  for  the  6th,  7th  and  8th  months  as  depicted  in  Table  7,  our 
model  proposes  the  reverse  as  seen  from  6th  and  7th  terms  of  assessed  OPM  values  as 
negative.  Finally,  discontinuities  are  experienced  while  the  relevant  terms’  CQTF’s  are 
applied  to  Equation  (5)  . 

In  conclusion,  CQTF-included  MM  is  not  validated  because  of  the  erroneous 
numbers  caused  by  the  use  of  continuous  function.  Thus,  the  MM  is  built  according  to  a 

25  Equation  (5)  became  discontinuous  between  6, and  6. 1759  at  7th  term  when  Equations(20)  and  (21)  were  applied 
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second  approach,  which  is  Discrete  Linear  Time  Functions  (DLTF)  included  MM.  Then 
the  findings  are  examined  in  order  to  attempt  to  validate  the  approach  and  interpret  the 
results  if  the  model  is  validated. 

Finding  the  Time  Functions  of  the  Metrics. 

1)  Building  Discrete  Linear  Time  Functions  (DLTF) 

a)  The  contractually-defined  DLTF’s  of  the  metrics  for  Case  1 

Both  metrics  are  defined  as  being  satisfied  by  the  constant  values  at  the  bottom- 
line  as  seen  in  Table  7.  Thus,  they  are  represented  as  depicted  in  Equations  (6)  and  (7)  for 
all  terms  without  changing. 

b)  The  assessed  DLTF’s  of  the  Metrics  for  Case  1 

First  of  all,  the  iterative  calculations  defined  in  Equations  (2),  (3)  and  (4)  are  used 
for  calculating  the  DLTF’s  for  each  metric.  The  relevant  calculations  are  made  from  the 
previous  term  to  the  latest  tenn  via  Matcad  14.  By  doing  so,  the  time  functions  for  the 
assessed  values  of  defined  metrics  are  revealed  for  relevant  terms.  The  calculations  of  the 
first  four  terms  are  depicted  in  Appendix  B.  Secondly,  the  relevant  time  functions  of  the 
terms  will  be  linear  and  they  are  depicted  from  Equation  (22),  to  Equation  (33). 

SMAal(t)  =  -26/  +  92. 1  (22) 

where 

SMAai  (t)  =  time  function  of  SMA  metric  for  the  1st  term 

AWCTaX(i)  =  0.2/  +  1.8  (23) 

where 

AWCTai  (t)  =  time  function  of  AWCT  metric  for  the  1st  tenn 
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SMAa2(t)  =  1.6/ +  89.5  (24) 

where 

SMAai  (t)  =  time  function  of  SMA  metric  for  the  2nd  tenn 

AWCTal(t)  =  0.9t  +  2  (25) 

where 

AWCTa2  (t)  =  time  function  of  AWCT  metric  for  the  2nd  term 

SMAa3(t)  -  -4.7/ +  91.1  (26) 

where 

SMAa3  (t)  =  time  function  of  SMA  metric  for  the  3rd  tenn 

AWCTa3(t)  =  l.2t  +  l.l  (27) 

where 

AWCTa3  (t)  =  time  function  of  AWCT  metric  for  the  3  rd  tenn 

SMAa4(t)  =  13.6f +  86.4  (28) 

where 

SMAa4  (t)  =  time  function  of  SMA  metric  for  the  4th  tenn 

AWCTa4(t)  =  -0.6f +  2.3  (29) 

where 

AWCTa4  (t)  =  time  function  of  AWCT  metric  for  the  4th  tenn 

SMAa5(t)  =  -2.5t  +  100  (30) 

where 

SMAa5  (t)  =  time  function  of  SMA  metric  for  the  5th  tenn 
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AWCTa5(t)  =  -0.\t  +  \.l  (31) 

where 

AWCTa5  (t)  =  time  function  of  AWCT  metric  for  the  5th  term 

SMAa6(t)  =  0.  If +  97.5  (32) 

where 

SMAa6  (t)  =  time  function  of  SMA  metric  for  the  6th  tenn 

AWCTa6(t)  =  0.2f +  1.6  (33) 

where 

AWCTa6  (t)  =  time  function  of  AWCT  metric  for  the  6th  tenn 

SMAal(t)  =  -1.2f +  97.6  (34) 

where 

SMAa7  (t)  =  time  function  of  SMA  metric  for  the  7th  term 

AWCTal(t)  =  0.\t  +  \.9  (35) 

where 

AWCTa7  (t)  =  time  function  of  AWCT  metric  for  the  7th  tenn 

Calculation  of  OPMs  for  DLTF-included  MM. 

Equation  (5)  is  used  to  calculate  the  OPMs  for  each  term  both  for  the  contractual 
and  the  assessed  performance  metrics,  but  this  time,  0  (zero)  and  l(one)  are  used  for  the 
boundaries  of  the  integrals,  because  of  using  DLTF’s.  The  area  of  interest  is  between  the 
lines  connecting  the  actual  and  assessed  values. 
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Table  10  organizes  the  calculated  OPMs  for  both  the  contract  and  the  assessed 
values.  The  table  refers  to  the  time  frame  between  two  consecutive  terms  of  Table  7.  For 
example  ,  the  1st  term  of  Table  10  covers  the  time  frame  between  S-04  and  0-04  of  Table 
7.  AOPMs  refer  to  the  difference  between  OPM  ASSESSED  and  OPM  CONTRACT  in 
the  table. 


Tal 

)le  10.  The  Calculated  Values  of  OPMs  via  DLTF 

TERMS 

OPM  CONTRACT 

OPM  ASSESSED 

A  OPM's 

1st 

42.5 

47.8 

5.334 

42.5 

60 

17.483 

3rd 

42.5 

54.6 

12.052 

4th 

42.5 

47 

4.454 

5th 

42.5 

59.9 

17.367 

6th 

42.5 

57.4 

14.949 

r^ttl 

42.5 

49.8 

7.254 

All 

283.863 

376 

78.893 

Analysis  of  DLTF-included  MM 

First  of  all,  DLTF-included  MM  is  validated  both  via  the  experts’  opinions  and 
via  the  results  gathered.  The  assessed  OPM  values  indicate  that  the  contractor  satisfied 
the  contractual  requirements  for  all  terms  by  means  of  OPMs.  However,  the  overall 
performance  of  the  contractor  has  a  waving  pattern  as  seen  from  the  OPM  ASSESSED 
values.  The  overall  performance  behavior  of  the  system  also  has  a  waving  pattern  as  seen 
from  the  values  of  AOPMs.  In  conclusion,  the  MM  adds  additional  insights  over  the 
insights  gained  from  the  CEM  and  the  IM  is  built  at  the  next  step. 
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Step  3:  Building  the  Integrated  Model  (IM) 

First  of  all,  the  data  of  Case  lare  processed  and  analyzed  at  the  individual  metric 
level  via  the  CEM  and  then  the  same  data  are  processed  and  analyzed  via  the  MM  to 
calculate  the  overall  performance  at  the  system  level.  The  next  step  is  building  the  IM  in 
order  to  organize  and  integrate  the  cumulated  results  of  the  CEM  and  the  MM  and  a 
dashboard  form  is  used  for  this  purpose  as  depicted  in  Table  11. 
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Table  11.  The  Dashboard  Form  of  Integrated  Model  (IM)  of  Case  1 


THE  INTEGRATED  MODEL  (IM) 

CEM 

MM 

Metrics 

SMA 

ACWT  (CONUS) 

OPM 

Months 

Bottom- 

line 

Actual- 

average 

Satisfied 

Bottom- 

line 

Actual- 

average 

Satisfied 

Months 

Terms 

OPM 

Contract 

OPM 

Assessed 

Satisfied 

A 

OPMs 

S-04 

1 

85% 

92.10% 

YES 

2  days 

1.8 

days 

YES 

S-04&0-04 

1 

42.5 

47.834 

YES 

5.334 

0-04 

2 

85% 

89.50% 

YES 

2  days 

2.0 

days 

YES 

0-04&N-04 

2 

42.5 

59.983 

YES 

17.483 

N-04 

3 

85% 

91.10% 

YES 

2  days 

1.1 

days 

YES 

N-04&D-04 

3 

42.5 

54.552 

YES 

12.052 

D-04 

4 

85% 

86.40% 

YES 

2  days 

2.3 

days 

NO 

D-04&J-04 

4 

42.5 

46.954 

YES 

4.454 

J-05 

5 

85% 

100.00% 

YES 

2  days 

1.7 

days 

YES 

J-05&F-05 

5 

42.5 

59.867 

YES 

17.367 

F-05 

6 

85% 

97.50% 

YES 

2  days 

1.6 

days 

YES 

F-05&M-05 

6 

42.5 

57.449 

YES 

14.949 

M-05 

7 

85% 

97.60% 

YES 

2  days 

1.8 

days 

YES 

M-05&A-05 

7 

42.5 

49.754 

YES 

7.254 

A-05 

8 

85% 

96.40% 

YES 

2  days 

1.9 

days 

YES 

ALL 

ALL 

297.5 

376.393 

YES 

78.893 
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Step  4:  Analysis  and  Interpretation  of  the  Findings 

First  of  all,  although  ACWT  (CONUS)  didn’t  satisfy  the  contractual  requirements 
according  to  D-04  (4th  month)  at  the  CEM,  the  OPM  values  satisfied  the  contractual 
requirements  according  to  the  contract  within  the  3rd  and  4th  tenns  at  the  MM.  However, 
this  shouldn’t  be  interpreted  as  favoring  the  contractor  despite  its  low  level  of 
performance.  On  the  contrary,  the  contractor  should  achieve  at  least  92%  for  the  SMA 
metric  and  1.9  days  for  the  AWCT  (CONUS)  metric  instead  of  85%  and  2  days 
respectively  for  the  4th  tenn  in  order  to  satisfy  the  contractually-mandated  OPM  of  42.5 
points  according  to  our  sensitivity  analysis  (See  Appendix  C). 

As  seen  from  this  example,  the  OPM  may  also  be  a  useful  tool  to  resolve  a 
potential  conflict  via  creating  a  win-win  situation  for  both  parties.  From  the 
Government’s  perspective,  although  the  contractual  limits  kept  fixed  for  the  relevant 
metrics,  the  OPM  enabled  the  limits  to  increase  dynamically  from  the  point  where  the 
conflict  was  experienced  to  the  next  month.  (4th  tenn  of  Table  10)  By  this  way,  a  chance 
is  given  to  the  contractor,  but  a  higher  performance  requirement  is  set  automatically. 
Thus,  the  contractor  is  motivated  for  a  higher  performance.  From  the  contractor’s 
perspective,  a  chance  is  given  to  him  to  prove  its  capabilities  within  one  tenn  without  a 
penalty. 

It  is  also  seen  that  the  contractor  is  struggling  to  set  his  perfonnance  at  a  high 
level.  Basically  there’s  a  waving  pattern  of  the  contractor’s  performance  and  the  overall 
performance  of  the  contractor  should  be  investigated  and  analyzed.  This  may  be  revealed 
by  comparing  the  contractual  overall  performance  metric  (OPM)  with  the  assessed  OPM 
by  the  contractor.  (See  Figure  15) 
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012345678 

Figure  14.  The  Comparison  of  Assessed  OPMs  with  Contractual  OPMs 


As  seen  in  the  figure,  the  contractor’s  overall  perfonnance  has  a  waving  pattern 
with  a  period  of  four  months,  in  which  the  first  two  months  have  an  increasing  behavior 
and  the  last  two  months  have  a  decreasing  behavior. 

Although  the  amount  of  the  data  may  not  be  enough  to  make  a  forecast  about  the 
following  terms,  it  may  be  expected  that  the  following  two  months  after  the  7th  month 
will  be  an  inclining  period  of  overall  performance  and  there  would  be  a  decrease  of 
performance  between  the  9th  and  1 1th  months.  The  lowest  points  of  this  waving  pattern 
are  very  close  to  the  contractually-defined  bottom  line  and  the  1 1th  month  may  be  very 
risky  for  both  parties  by  means  of  satisfying  the  contractual  requirements.  Figure  16 
depicts  the  overall  performance  trend  of  the  system  via  considering  the  differences  of  the 
assessed  OPMs  and  contractual  OPMs. 
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Figure  15.  The  Performance  Trend  of  the  System 


First  of  all,  the  overall  performance  of  the  system  is  not  stabilized.  It  peaks  at  the 
2nd  term  and  the  bottom  point  is  at  the  4th  term.  Secondly,  the  difference  of  the  bottom 
and  the  peak  performance  points  of  the  system  are  nearly  14  points.  Finally,  the  bottom 
point  is  nearly  4  points  and  it  is  very  close  to  the  zero  level  performance  which  is  the 
border  of  satisfaction  and  dissatisfaction  level  of  the  system. 
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Case  2 


C-17  Globemaster  Sustainment  Program; 

In  this  case  there  are  five  metrics  that  can  be  examined  using  the  4SPEM  and  the 
following  section  is  taken  from  Mahon,  2007  until  the  first  step  of  the  4SPEM. 

1 .  The  desired  performance  outcomes  are  mentioned  in  the  PBL  contract  as: 

Globemaster  Sustainment  Aircraft  Availability  (GSAA):  The  required 
performance  in  this  measure  is  to  maximize  the  number  of  aircraft  available 
for  missions.  The  requirement  equals  the  percentage  of  mission  capable 
aircraft  relative  to  total  fleet  assigned.  It  is  a  monthly  measurement  and  is 
adjusted  to  account  for  aircraft  undergoing  modification.  The  measure 
provides  a  90-day  period  following  surge  operations  during  which  Boeing  is 
relieved  of  meeting  the  performance  measurement  if  the  metric  variance  is 
five  percent  less  than  the  previous  12  months  average  as  measured  prior  to  the 
surge.  The  annual  performance  will  be  the  12-month  average  of  the  adjusted 
GSAA  requirement  (Mahon,  2007,  p  62). 

Flying  Hours  Achievable  (FHA):  The  required  perfonnance  in  this  measure  is 
to  maintain  the  highest  level  of  flying  hours  available  for  wartime  mission 
availability  based  upon  the  Weapon  System  Management  Infonnation 
System-  Sustainability  Assessment  Module  (WSMIS-SAM).  The 
measurement  is  the  percent  of  wartime  flying  hours  achieved  given  spares 
availability  compared  to  the  number  of  required  hours  as  compiled  by 
WSMIS-SAM.  It  is  measured  the  first  week  of  the  month.  Upon  receipt  of 
readiness  spares  packages  (RSP),  Boeing  is  given  a  60-day  grace  period  to 
reconcile  the  RSP,  during  which  the  measurement  is  not  counted  (Mahon, 
2007,  p  63). 

Mission  Capable  (MICAP):  The  required  performance  in  this  measure  is  to 
deliver  MICAP  assets  within  48  hours  in  the  continental  United  States 
(CONUS)  and  within  96  hours  in  the  FMS  United  Kingdom.  The 
measurement  is  the  percent  of  MICAP  events  successfully  fulfilled  within  the 
48  hours  CONUS  or  96  hours  UK  compared  to  the  actual  number  of  MICAP 
events.  The  event  starts  when  the  Government  requests  MICAP  services  and 
documents  it  in  a  MICAP  log  and  ends  when  the  MICAP  item  is  received  at 
the  end  destination  and  entered  into  the  Standard  Base  Supply  System 
(Mahon,  2007,  p  63). 

Aircraft  Depot  Maintenance  Scheduling  Effectiveness  (ADMSE):  The 
required  performance  in  this  measure  is  completion  of  scheduled  maintenance 
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tasks  to  include  negotiated  and  approved  work,  including  over  and  above 
work  requirements,  within  the  negotiated  schedule  time.  Perfonnance  is 
calculated  by  dividing  the  number  of  days  taken  to  complete  scheduled 
maintenance  tasks  by  the  negotiated  number  of  days.  The  goal  is  to  maximize 
the  amount  of  work  done  within  the  negotiated  time.  (Mahon,  2007,  p  63) 


Issue  Effectiveness  (IE):  It  represents  the  number  of  issue  requests  filled 
compared  to  the  number  received  and  is  a  monthly  measure.  Issue 
effectiveness  at  new  bed  down  bases  will  not  be  counted  in  overall  issue 
effectiveness  during  the  time  aircraft  are  being  delivered.  This  measure  has 
incentives  to  forecast  demand  rather  than  rely  on  past  demands.  (Mahon, 
2007,  p  63) 


Customer  Satisfaction  (CS):  The  required  perfonnance  in  this  measure  is 
customer  satisfaction.  Using  a  concept  called  the  shared  destiny/operating 
principles,  the  customer  scores  1 1  focus  areas  of  the  perfonnance  work 
statement  across  six  criteria  .Administered  quarterly,  the  survey  coincides 
with  the  award-fee  schedule.  The  survey  is  distributed  to  the  C-17  Special 
Program  Office,  C-17  system  support  manager,  using  commands  and  the 
Defense  Contract  Management  Agency.  The  rating  is  detennined  by 
averaging  scores  in  each  focus  area  and  then  averaging  all  areas  to  obtain  a 
single  rating.  (Mahon,  2007,  p  63) 

The  contractually-defined  requirements  for  the  Case  2  are  organized  and  depicted 
in  Table  12. 


Table  12.  Contractual  Requirements  for  Case  2 

_ (Adapted  from  Mahon,  2007) _ 


REQUIRED 
PERFORMANCE  FOR 
RELATED  METRICS 

FISCAL  YEA] 

RS 

FY04 

FY05 

FY06 

FY07 

FY08 

GSAAlb 

69 

71.7 

74.2 

78.4 

79.1 

FHA 

95 

95 

95 

95 

95 

MICAP 

80 

80 

80 

80 

80 

ADMSE 

98-100 

98-100 

98-100 

98-100 

98-100 

IE  (XD  items) 

82 

82 

82 

82 

82 

IE  (All  other  items) 

67 

75 

78 

80 

80 

CS 

2-5 

2-5 

2-5 

2-5 

2-5 

26  The  contractual  requirement  of  GSAA  for  the  first  three  periods  were:  75.0%  ,  76.3%,  77.6%  and  they  were  revised  as  it  is 
mentioned  in  Table  13. 
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3.  The  achieved  performance  levels  of  the  first  three  years  are  depicted  in  Table  13. 


Table  13.  Achieved  Average  Performance  Results  for  Case  2 

_ (Adapted  from  Mahon,  2008) _ 


ACHIEVED  PERFORMANCE 
FOR  RELATED  METRICS 

FISCAL  YEARS 

FY04 

FY05 

FY06 

GSAA27 

74.1 

73.7 

74.4 

FHA 

97.3 

98.9 

99.2 

MICAP 

90.2 

96.1 

91.5 

ADMSE 

100.06 

100.08 

100.4 

IE 

76.5 

76.2 

79 

CS 

N/A 

N/A 

N/A 

According  to  the  data  gathered  from  Mahon,  2008  we  can  start  to  build  the 
4SPEM  of  Case  2. 


the  4SPEM  for  Case  2 

Step  1:  Building  the  Current  Evaluation  Model  (CEM). 

First  of  all  the  GSAA,  FHA,  MICAP,  ADMSE  and  IE  are  used  in  the  CEM. 
Second,  for  GSAA  the  adjusted  requirement,  for  ADMSE  the  minimum  range  and  for  the 
rest  of  the  metrics  the  requirements  are  be  used  as  bottom-line  constraints. 

Secondly,  the  bottom-line  of  each  individual  metric  is  defined  according  to  the 
contract  and  then  the  assessed  values  of  these  metrics  are  checked  for  the  relevant  terms 
in  CEM.  The  CEM  of  the  C-17  program  is  depicted  in  Table  14. 

27  Eighty-five  percent  of  the  award  fee  is  based  upon  the  GSAA  performance  measurement. 
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Table  14.  The  CEM  of  C-17  Globemaster 


Fiscal 

Y  ears 

FY04 

FY05 

FY06 

Metrics 

Bottom- 

line 

Actual- 

Annual 

average 

Satis¬ 

fied 

Bottom- 

line 

Actual- 

Annual 

average 

Satis¬ 

fied 

Bottom- 

line 

Actual- 

Annual 

average 

Satis¬ 

fied 

GSAA 

69.0 

74.1 

YES 

71.7 

73.7 

YES 

74.2 

74.4 

YES 

FHA 

95 

97.3 

YES 

95 

98.9 

YES 

95 

99.2 

YES 

MICAP 

80 

90.2 

YES 

80 

96.1 

YES 

80 

91.5 

YES 

ADMSE 

98 

100.06 

YES 

98 

100.08 

YES 

98 

100.4 

YES 

IE 

67 

76.5 

YES 

75 

76.2 

YES 

78 

79 

YES 

As  seen  from  the  CEM,  each  one  of  the  performance  metrics  satisfied  the 
contractual  requirements  individually.  However,  some  of  the  assessed  values  of  the 
metrics  are  plateauing  such  as  GSAA,  some  of  them  have  an  upward  trend  such  as  FHA 
and  ADMSE,  and  some  of  them  have  waving  patterns  such  as  MICAP  and  IE. 
Additionally,  the  overall  performance  trend  of  the  system  is  hard  to  reveal. 

Step  2:  Building  the  Mathematical  Model  (MM). 

Building  the  Customized  Integration  Formulas  (CIF’s)  for  Case  2. 

First  of  all,  the  effects  of  each  metric  should  be  understood  from  the  system’s 
perspective.  However,  it  is  not  necessary  to  calculate  the  correlations  of  the  metrics. 
Defining  an  individual  metric  as  bigger  is  better  (+  effect),  or  smaller  is  better  (-  effect), 
is  enough  to  build  the  integration  fonnula.  Table  15  depicts  the  defined  effects  of  the 
metrics  according  to  the  contract. 
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Table  15.  The  Effects  of  the  metrics 


Metrics 

The  bigger  is  better? 

The  effect  of  the  metric 

GSAA 

YES 

(+) 

FHA 

YES 

(+) 

MICAP 

YES 

(+) 

ADMSE 

YES 

(+) 

IE 

YES 

(+) 

Secondly,  all  of  the  defined  metrics  have  positive  effects  on  the  system  according 
to  the  gathered  data.  Although  it’s  out  of  the  scope  of  this  study,  it  may  be  argued  to 
define  only  positive  effect  metrics  in  the  contract  at  Pre-PBL  phase  may  be  a  concern. 
One  or  more  negative  effect  metrics  should  be  defined  in  any  kinds  of  PBL  contracts  to 
avoid  the  “more  is  better”  only  mindset. 

Finally,  the  CIF  is  built  according  to  defined  effects  of  the  metrics  for  Case  2 
using  Equation  (1)  and  it  is  depicted  in  Equation  (36). 


n  n  n  g  n 

OPM  =  J  GSAA(t).dt  x  J FHA(t).dt  x  J  MICAP(t  ).dt  x  j  ADMSE(t).dt  x  J  IE{t).dt 


where 

GSAA(t) 

FHA(t) 

MICAP(t) 

ADMSE(t) 

IE(t) 

g 

h 


=  the  relevant  time  function  of  GSAA  metric 
=  the  relevant  time  function  of  FHA  metric 
=  relevant  time  function  of  MIC  AP  metric 
=  relevant  time  function  of  ADMSE  metric 
=  relevant  time  function  of  IE  metric 
=  the  beginning  of  the  term 
=  the  end  of  the  tenn 


(36) 
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Finding  the  Time  Functions  of  the  Metrics. 

1)  Building  Discrete  Linear  Time  Functions  (DLTF) 

First  of  all,  building  the  MM  via  discrete  linear  time  functions  (DLTF’s)  was 
shown  in  Case  1 .  Thus,  the  iterative  calculations  which  were  defined  in  Equations  (2),  (3) 
and  (4)  are  used  for  calculating  the  DLTF’s  for  each  metric  from  the  previous  tenn  to  the 
latest  tenn  to  reveal  the  time  functions  for  both  the  contractually-defined  and  the  assessed 
values  of  the  metrics  for  relevant  tenns.  (See  Appendix  D) 

Secondly,  the  relevant  time  functions  of  the  terms  are  linear  and  they  are  depicted 
from  Equation  (37),  to  Equation  (56). 

a)  The  Contractually-defined  DLTF’s  of  the  Metrics  for  Case  2 

The  contractually-defined  time  functions  of  the  relevant  metrics  for  the  1st  and  the 
2nd  tenns  are  depicted  from  Equation  (37)  to  Equation  (46). 

GSAAcl(t)  =  2.7 1  +  69  (37) 

where 

GSAAci  (t)  =  the  time  function  of  GSAA  metric  for  the  1st  term 

FHAJt)  =  95  (38) 

where 

FHAci  (t)  =  the  time  function  of  FHA  metric  for  the  1st  tenn 

MICAPcl(t)  =  80  (39) 

where 

MICAP  ci  (t)  =  the  time  function  of  MICAP  metric  for  the  lbt  tenn 

ADMSE cX(t)  =  98  (40) 

where 

ADMSE  ci  (t)  =  the  time  function  of  ADMSE  metric  for  the  1st  term 
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where 


IEcX{t)  =  8^  +  67 


(41) 


IE  ci  (t)  =  the  time  function  of  IE  metric  for  the  1st  tenn 


GSAAc2  (t)  =  2.5?  +  7 1 .7  (42) 

where 

GSAAC2  (t)  =  the  time  function  of  GSAA  metric  for  the  2nd  tenn 


FHAJt)  =  95  (43) 

where 

FHAC2  (t)  =  the  time  function  of  FHA  metric  for  the  2nd  term 


MICAPcl{t)  =  80  (44) 

where 

MIC  AP  C2  (t)  =  the  time  function  of  MIC  AP  metric  for  the  2nd  tenn 


ADMSE Jt)  =  98  (45) 

where 

ADMSE  C2  (t)  =  the  time  function  of  ADMSE  metric  for  the  2nd  tenn 


IEc2(t)  =  3t  +  15  (46) 

where 

IE  C2  (t)  =  the  time  function  of  IE  metric  for  the  2nd  term 


b)  The  Assessed  DLTF’s  of  the  Metrics  for  Case  2 

The  assessed  time  functions  of  the  relevant  metrics  for  the  1st  and  the  2nd  terms 
are  depicted  from  Equation  (47)  to  Equation  (56). 

GSAAal  (t)  =  -4t  +  74. 1  (47) 

where 

GSAAai  (t)  =  the  time  function  of  GSAA  metric  for  the  1st  term 
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FHAaX(t)  =  1.6/  +  97.3  (48) 

where 

FHAai  (t)  =  the  time  function  of  FHA  metric  for  the  1st  tenn 


MICAPJt)  =  5.9 1  +  90.2  (49) 

where 

MICAP  ai  (t)  =  the  time  function  of  MICAP  metric  for  the  1st  tenn 


ADMSE al  (i t )  =  02^  +  1 00.6  (50) 

where 

ADMSE  ai  (t)  =  the  time  function  of  ADMSE  metric  for  the  1st  term 


IEal(t)  =  -0.3^  +  76.5  (51) 

where 

IE  ai  (t)  =  the  time  function  of  IE  metric  for  the  1st  tenn 


GSAAa2(t)  =  0.7 1  +  13.1  (52) 

where 

GSAAa2  (t)  =  the  time  function  of  GSAA  metric  for  the  2nd  tenn 


FHAa2(t)  =  0.3? +  98.9  (53) 

where 

FHAa2  (t)  =  the  time  function  of  FHA  metric  for  the  2nd  term 


MICAPa2  (t)  =  -4.6 1  +  96. 1  (54) 

where 

MICAP  a2  (0  =  the  time  function  of  MICAP  metric  for  the  2nd  tenn 


ADMSE  a2  ( t )  =  0.32f  + 1 00.08  (55) 

where 

ADMSE  a2  (0  =  the  time  function  of  ADMSE  metric  for  the  2nd  tenn 
IEa2(t)  =  2.St  +  76.2  (56) 

where 

IE  a2  (0  =  the  time  function  of  IE  metric  for  the  2nd  term 
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Calculation  of  OPMs  for  Case  2. 


Since  we  calculated  all  the  necessary  time  functions  of  the  relevant  metrics,  we 
will  use  Equation  (36)  to  calculate  the  OPMs  for  each  term  both  for  the  contractual  and 
the  assessed  performance  metrics. 

Fiscal  year  (FY)  2004  is  accepted  as  the  origin  time  (time  zero)  in  our 
calculations-  and  Table  16  organizes  the  calculated  OPMs  for  both  the  contractual  and 
the  assessed  values.  The  terms  in  the  table  refer  to  the  time  frame  between  two 
consecutive  terms  of  Table  14.  AOPMs  refer  to  the  differences  between  OPM 
ASSESSED  and  OPM  CONTRACT  in  the  table. 


Table  16.  The  Calculated  Values  of  OPMs 


TERMS 

OPM  CONTRACT 
(xlO9) 

OPM  ASSESSED 
(xlO9) 

A  OPM’s 
(xlO9) 

1st 

3.72 

5.192 

1.472 

2nd 

4.156 

5.352 

1.196 

All 

7.876 

10.544 

2.668 

Analysis  of  the  MM  for  Case  2 

First  of  all,  the  contractually  demanded  overall  perfonnance  requirements  were 
satisfied  for  both  tenns.  However,  the  overall  performance  of  the  contractor  has  a 
plateauing  trend  as  seen  from  the  OPM  ASSESSED  values.  A  declining  trend  is  also  seen 
for  the  overall  perfonnance  behavior  of  the  system  via  the  values  of  AOPMs. 


FY  2004  =  t]=0,  FY2005  =  t2=l ,  FY  2006  =  t3  =  2 
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Secondly,  it  is  revealed  that  the  Government  set  the  OPM  low  for  the  first  tenn 
and  the  contractor  achieved  nearly  40%  higher  OPM.  The  Government’s  OPM 
expectation  increased  1 1.72%  for  the  second  term,  but  the  contractor’s  perfonnance  of 
achieving  OPM  stayed  very  low  relative  to  that  expectation  and  increased  only  3%  for  the 
second  term. 

Step  3:  Building  the  Integrated  Model  (IM) 

First  of  all  the  data  of  Case  2  are  processed  and  analyzed  at  the  individual  metric 
level  via  the  CEM  and  then  the  same  data  are  processed  and  analyzed  via  the  MM  to 
calculate  the  overall  performance  at  the  system  level.  The  next  step  is  building  the  IM  in 
order  to  organize  and  integrate  the  cumulated  results  of  the  CEM  and  the  MM  and  a 
dashboard  form  is  used  for  this  purpose  as  depicted  in  Table  17. 

Finally,  the  overall  perfonnance  trend  of  the  contractor  and  the  overall 
performance  trend  of  the  system  are  investigated  and  analyzed.  The  amount  of  the  data 
may  seem  not  enough  to  make  a  forecast;  however,  the  data  belong  to  already  processed 
annual  terms  which  includes  36  monthly  terms.  Table  17  organizes  and  integrates  the 
cumulated  results  of  the  CEM  and  the  MM  and  represents  the  main  body  of  the  IM. 
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Table  17.  The  Dashboard  Form  of  Integrated  Model  (IM)  of  Case  2 


THE  INTEGRATED  MODEL  (IM) 

CEM 

Fiscal  Years 

FY04 

FY05 

FY06 

Metrics 

Bottom-line 

Actual- 

Annual 

average 

Satisfied 

Bottom- 

line 

Actual- 

Annual 

average 

Satisfied 

Bottom- 

line 

Actual- 

Annual 

average 

Satisfied 

GSAA 

69 

74.1 

YES 

71.7 

73.7 

YES 

74.2 

74.4 

YES 

FHA 

95 

97.3 

YES 

95 

98.9 

YES 

95 

99.2 

YES 

MICAP 

80 

90.2 

YES 

80 

96.1 

YES 

80 

91.5 

YES 

ADMSE 

98 

100.06 

YES 

98 

100.08 

YES 

98 

100.4 

YES 

IE 

67 

76.5 

YES 

75 

76.2 

YES 

78 

79 

YES 

MM 

OPM 

Fiscal  Years 

Terms 

OPM  Contract 

OPM  Assessed 

Satisfied 

AOPMs 

FY04-FY05 

1 

3.72 

5.192 

YES 

1.472 

FY05-FY06 

2 

4.156 

5.352 

YES 

1.196 

FY04-FY06 

ALL 

7.8"6 

10.544 

YES 

2.668 

Step  4:  Analysis  and  Interpretation  of  the  Findings 

First  of  all,  the  contractor  satisfied  all  the  contractual  requirements  both  at  the 
individual  and  at  the  overall  metric  level.  Although  each  assessed  value  of  the  individual 
metrics  has  a  different  trend  at  every  term,  the  assessed  values  of  the  OPMs  have  a  very 
stable  trend  for  the  relevant  terms. 

Secondly,  there’s  no  conflict  experienced  so  far;  however,  the  contractor’s  overall 
performance  trend  is  plateauing  while  time  passes.  On  the  contrary,  the  overall 
performance  expectanctions  of  the  Government  is  increasing;  therefore,  a  conflict  may  be 
experienced  in  the  following  terms  when  the  Government’s  increasing  expectations 
became  infeasible  for  the  contractor. 

Furthennore,  the  success  of  the  contractor  may  be  arguable  for  the  first  term. 
Because  according  to  a  basic  comparison  between  the  contractually-defined  OPM  and  the 
assessed  value  of  the  OPM  for  the  first  term,  the  bottom-line  OPM  may  be  set  lower  than 
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necessary  for  the  first  term  by  the  Government,  and  the  contractor  may  pick  the  lower 
hanging  fruit. 

Figure  17  depicts  the  comparison  of  the  contractually-defined  OPMs  with  the 
assessed  values  of  the  OPMs. 


Figure  16.  The  Comparison  of  Assessed  OPMs  with  Contractual  OPMs 

As  seen  from  the  figure,  the  contractor’s  overall  performance  has  a  slightly 
upward  trend,  but  the  contractual  expectation  for  the  overall  perfonnance  has  a  relatively 
higher  upward  trend.  The  slope  of  the  ASSESSED  function  in  the  figure  is  0.16; 
however,  the  slope  of  the  CONTRACT  is  0.436  which  is  nearly  2.725  times  greater  than 
the  ASSESSED. 

Briefly,  the  contractual  expectation  increases  2.725  units  for  each  unit  increment 
of  contractor’s  achievement  so  far.  This  supports  our  proposal  of  expectation  for  a 
conflict  in  the  near  future  between  the  Government  and  the  contractor  for  this  case. 
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Additionally,  the  overall  perfonnance  trend  of  the  system  should  also  be  taken 
into  consideration.  Figure  18  reveals  this  trend  via  the  reflecting  the  differences  of  the 
assessed  OPMs  and  contractual  OPMs. 


0  12  3 

Figure  17.  The  Performance  Trend  of  the  System 


The  overall  performance  trend  of  the  system  has  a  downward  trend  and  the  slope 
of  this  function  is  -0.276.  It  may  be  proposed  that  the  system  is  heading  to  the  zero  level 
performance  difference  which  is  the  border  of  satisfaction  level. 


Conclusion 

First  of  all,  the  4SPEM  is  applied  to  two  real  life  cases  to  investigate  whether  or 
not  it  could  reveal  additional  insights  for  the  decision  makers.  Each  one  of  the  cases  has 
different  properties  and  both  of  them  are  benefit  from  assess  the  model. 
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Case  1  has  both  positive  and  negative  effect  metrics  and  two  main  problems  were 
experienced  which  are:  discontinuity  effects  and  huge  fluctuations  while  applying  the 
Continuous  Quadratic  Time  Functions  (CQTF’s)  included  MM.  Further  investigation 
revealed  that  the  relevant  continuous  time  functions  became  very  problematic,  especially 
the  ones  that  belong  to  the  negative  effect  metrics,  while  applying  the  CIF  because  of 
high  quadratic  terms  and  increasing  values  of  coefficients.  Therefore,  the  4SPEM  is 
upgraded,  and  the  Discrete  Linear  Time  Functions  (DLTF’s)  approach  is  used. 
Additionally,  a  conflict  situation  is  investigated  and  a  reasonable  resolution  for  that 
situation  is  proposed  via  the  4SPEM.  After  examining  the  model  results,  it  is  understood 
that  it  may  be  applicable  to  other  cases. 

Case  2  has  many  different  metrics  and  the  applicability  of  the  model  is  tested  for 
many-metrics  situation.  Additionally,  there  is  an  increasing  set  of  contractual 
requirements  in  the  case.  The  Government’s  decision  of  setting  an  upward  trend  for  the 
minimum  level  of  satisfaction  for  the  selected  metrics  may  imply  two  issues.  First,  there 
may  be  conflicts  between  the  assessed  values  of  the  metrics  at  the  individual  level  if  the 
contractor  finds  it  easier  to  play  on  some  metrics  rather  than  all.  Second,  demanding  an 
incremental  trend  for  individual  metrics  may  be  demanding  an  infeasible  overall 
performance  from  the  contractor  eventually. 

Finally,  we  could  analyze  the  overall  performance  patterns  of  both  the  contractor 
and  the  system  for  the  short  run.  Furthermore  the  4SPEM  enabled  us  to  make  forecasts  as 
well  as  gain  additional  insights  for  the  long  run. 
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V.  Conclusions 


Introduction 

First  of  all,  PBL  is  a  relatively  new  and  challenging  concept  and  it  was  created  to 
satisfy  the  highly-elastic  needs  of  the  war  fighter  through  the  GWOT.  The  main  promise 
of  the  PBL  is:  enabling  and  sustaining  a  desired  level  of  performance  with  a  reasonable 
cost  via  the  innovative  approaches  and  the  cumulated  experiences  of  the  contractors. 

However,  this  study  revealed  that  the  promised  benefits  are  not  always  gathered 
by  the  Government  for  all  cases.  Two  possible  causes  of  this  situation  may  be 
inappropriately-defined  metrics  and  incapability  of  evaluation  processes. 

In  this  study  a  holistic  approach  is  adopted  and  the  post-PBL  period  is 
concentrated  on,  to  contribute  to  the  area  of  evaluation  processes.  Therefore,  the  4-Step 
Process  Evaluation  Model  (the  4SPEM)  is  proposed  for  evaluation  of  the  defined  metrics 
to  monitor  the  system  and  to  make  healthy  decisions  about  it.  Then  the  model  is 
examined  via  applying  it  to  two  selected  real  life  cases. 

The  research  revealed  that  the  4SPEM  may  be  a  very  useful  tool  for  the  decision 
makers.  First,  it  can  be  applicable  to  different  kinds  of  PBL  contracts.  Second,  it  helps  to 
gain  additional  insights  about  the  performance  of  the  system  both  at  the  individual 
metrics  level  and  at  the  overall  system  level.  Third,  it  may  reveal  the  possible  future 
performance  patterns  of  both  the  contractors  and  the  system.  Finally,  it  may  offer  a 
possible  resolution  of  a  conflict  situation  to  both  parties  of  a  PBL  contract. 
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In  conclusion,  the  core  objectives  which  are:  simplicity,  reasonability,  elasticity, 


precision,  capability,  integrality  and  objectivity  are  fulfilled  while  proposing  the  4- 
SPEM. 


The  Gathered  Answers  to  Investigative  Questions 

1)  What  are  the  general  metrics  defined  in  the  contracts? 

Actually,  there’s  neither  a  standard  template  for  PBL  contracts,  nor  standardized 
metrics.  However,  relevant  to  their  knowledge  about  PBL,  the  Defense  Acquisition 
University  (DAU)  mentioned  some  useful  metrics  as:  Operational  Availability  (Ao), 
Operational  Reliability  (R),  Cost  per  Unit  Usage  (CUU),  Logistics  Footprint  (LF)  and 
Logistics  Response  Time  (LRT)  in  their  published  book  of  “Performance  Based 
Logistics:  A  Program  Manager's  Product  Support  Guide”,  also  known  as  PBL  Guide. 
(DAU,  2005) 

However,  some  academics  mentioned  that  these  metrics  are  high  level  composite 
metrics  (Kumar  et  ah,  2007).  So  in  each  PBL  contract,  some  or  all  of  these  metrics  should 
be  tailored  according  to  the  desired  goals  and  would  be  negotiated  with  the  contractor/s. 

2)  What  is  the  general  current  evaluation  method  of  the  metrics? 

According  to  the  experts’  opinions,  in  most  of  the  PBL  contracts,  the  metrics  are 
evaluated  by  a  two  step  process  that  is: 

a)  Collecting  assessed  values  of  the  metrics  within  the  contractually-defined  terms, 
(ex:  average  monthly  values) 
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b)  Comparing  assessed  values  of  the  metrics  with  the  contractually  targeted  numbers 

9Q 

of  those  metrics  individually." 

3)  What  are  the  gaps  of  the  current  evaluation  method? 

a)  According  to  the  laws  of  statistics,  each  one  of  the  pair  wise  comparisons  includes 
risks  of  making  type  1  and  type  2  errors,  which  are  the  probabilities  of  making 
misjudgments  depending  on  a  statistical  comparison.  The  probability  of  making  a 
mistake  in  comparisons  increases  as  the  number  of  pairs  compared  increases. 

b)  According  to  the  Theory  of  Constraints;  local  improvements  do  not  necessarily 
mean  that  the  whole  system  improved  (Goldratt  and  Cox,  1992).  By  monitoring 
each  metric  individually,  we  may  not  visualize  the  big  picture  and  make  healthy 
decisions  at  a  strategic  level. 

c)  The  current  evaluation  method  may  be  useful  in  order  to  understand  if  the 
assessed  values  of  the  individual  metrics  satisfied  the  contractual  agreements. 
However,  as  some  academics  mentioned,  there  may  be  conflicts  between  the 
metrics  such  as:  “some  metrics  may  be  over  performing  while  the  others  are 
underperforming”  (Sols  et  ah,  2007). 

d)  “The  scarcest  resource  in  an  organization  is  a  manager’s  time”  (Kaplan  and 
Cooper.  1997,  p.144).  The  evaluation  of  the  metrics  individually  may  consume 
more  time  than  necessary  especially  for  the  senior  executives  for  two  reasons. 
First,  there  may  be  an  excessive  amount  of  defined  parameters  and  the  decision 
maker  may  need  time  to  understand  the  impact  of  each  one  of  them  on  the  system. 


29  In  addition  to  these  two  processes,  some  experts  mentioned  in  some  cases  a  process  of  weighing  the  parameters  and  calculating  a 
single  number  for  strategic  decisions.  This  additional  step  to  the  current  evaluation  method  is  mentioned  under  the  subject  heading  of 
“Evaluation  Process  of  PBL”  at  the  end  of  the  Literature  Review. 
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Second,  there  may  be  conflicts  between  parameters.  Although  the  decision  maker 
may  spend  much  time  and  effort,  s/he  may  not  use  the  performance  data 
appropriately  in  the  decision-making  process  (US  GAO  July  2008,  Highlights). 

4)  How  should  the  metrics  be  evaluated  to  improve  decision  making? 

First,  the  assessed  values  of  the  individual  metrics  should  be  evaluated  by  means 
of  satisfying  the  contractual  requirements  at  the  bottom-line.  Second,  a  further  step  would 
be  taken  and  the  system  should  be  evaluated  as  a  whole.  Third,  the  gathered  results  of  the 
evaluations  should  be  organized  and  integrated.  Finally,  the  results  should  be  analyzed 
and  the  interpreted. 

The  4SPEM  is  built  according  to  the  four  logical  steps  of  evaluation  which  were 
mentioned  earlier.  The  Current  Evaluation  Model  (CEM)  is  built  for  the  first  step  and 
checked  whether  or  not  the  metrics  were  satisfied  in  each  time  period.  Then,  the 
Mathematical  Model  (MM)  is  built  in  order  to  calculate  a  single  value  representing  the 
overall  performance  of  the  system  called  the  Overall  Perfonnance  Metric  (OPM)  and  the 
model  analyzed  the  calculated  OPMs  for  the  relevant  terms.  Next  the  Integrated  Model 
(IM)  is  built  to  organize  and  integrate  the  findings  of  the  CEM  and  the  MM. 

Finally,  the  cumulated  knowledge  gathered  from  IM  is  analyzed  and  the  results 
are  interpreted  as  the  fourth  step. 

5)  What  should  be  the  core  objectives  of  an  evaluation  model? 

The  reviewed  literature  revealed  that  some  models  were  introduced  for  the 
evaluation  processes  in  PBL  and  some  of  the  models  are  mentioned  in  the  current  study. 
Additionally  the  4-SPEM  is  developed  throughout  the  current  study  and  according  to  the 


91 


cumulated  knowledge.  The  objectives  of  a  model  which  would  be  used  in  evaluation 
processes  may  be  proposed  as: 


•  Simplicity  :  The  model  must  be  easy-to-understand  and  apply  by  the  decision 
makers. 

•  Reasonability:  The  assumptions  and  the  results  of  a  model  must  make  sense  to  the 
decision  maker. 

•  Elasticity  :  A  model  should  be  dynamic  and  can  be  applied  any  kinds  of  PBL 
contracts. 

•  Precision  :  The  results  should  be  precise  to  guide  the  decision  maker. 

•  Capability  :  A  model  should  be  capable  of  processing  any  number  of  metrics 

within  many  time  periods. 

•  Integrality  :  A  model  should  be  capable  of  integrating  all  metrics  and  calculate 
a  single  value  as  the  overall  performance. 

•  Objectivity  :  The  data  processing  and  the  outcomes  of  a  model  should  be  the 
same,  no  matter  who  performs  them.  Therefore,  the  model  should  be  quantitative 
and  relatively  closed  to  subjective  comments. 

6)  Can  the  proposed  model  in  this  thesis  be  used  for  forecasting,  or  building  new  PBL 

contracts? 

According  to  the  scope  of  this  thesis,  the  offered  approach  is  reactive,  so  at  the 
very  beginning  of  a  PBL  contract  concerned  with  a  brand  new  system,  this  approach  may 
not  be  an  effective  tool  for  forecasting. 


92 


However,  while  the  data  are  cumulatively  gathered  according  to  the  metrics 
defined  in  the  contract,  the  4SPEM  would  be  a  highly  effective  tool  in  order  to  monitor 
the  overall  performance  pattern  of  the  contractor  and  the  PBL-governed  system. 

7)  At  which  decision  level  may  the  proposed  model  be  used? 

First  of  all,  this  approach  may  be  used  both  for  operational-level  decisions  such 
as:  reward  and  penalty  decisions  within  a  contract  and  at  the  strategic-level  such  as 
extending  or  tenninating  the  contractual  agreement  with  the  same  contractor  or  making  a 
new  contract  about  the  same  system  with  another  contractor. 

Secondly,  the  4SPEM  may  be  used  to  resolve  the  conflicts  between  the 
Government  and  the  contractor  if  the  individual  metrics  of  a  term  conflicts  with  each 
other,  such  as  one  metric  is  above  the  defined  limits  whereas  another  is  below. 

Finally,  the  sub-components  of  the  4SPEM  may  be  useful  for  specific  evaluation 
needs  within  a  contract.  For  example,  we  may  use  the  MM  in  the  process  of  calculating  a 
single  number  for  composite  metrics  such  as  the  Logistics  Foot  Print  (LFP). 

Relevant  Findings  and  Recommendations 

First  of  all,  the  definition  of  metrics  plays  an  important  role  in  the  successful 
implementation  of  PBL.  Since  the  aim  is  satisfying  the  war  fighter’s  needs,  the 
operational  units  at  the  customer  side  should  have  the  responsibility  and  authority  to 
define  specifically  what  they  need.  Then  the  supporting  units  should  tailor  the  metrics 
with  a  cause  and  effect  relationship  to  fulfill  the  operational  needs.  If  any  of  the 
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individual  metrics  could  not  be  satisfied,  the  customer  should  penalize  the  contractor  and 
even  terminate  the  contract  if  necessary. 

Secondly,  the  evaluation  processes  should  be  determined  and  customized 
according  to  the  defined  metrics.  Although  developing  an  evaluation  process  prior  to 
defining  the  metrics  may  be  perceived  as  easier  and  pragmatic,  metrics  should  not  be 
defined  according  to  an  evaluation  process. 

Third,  subjective  approaches  such  as  weighting  of  metrics,  or  nonnalization  of 
evaluation  processes  are  not  recommended  at  this  time.  Basically,  favoring  one  metric 
over  another,  or  making  misassumptions  about  the  limits  of  the  metrics  may  lead  both  the 
Government  and  the  contractor  in  the  wrong  direction. 

All  the  metrics  should  be  perceived  as:  the  defined  needs  of  the  war  fighter,  thus, 
none  of  the  metrics  should  be  treated  differently  than  the  others. 

Finally,  an  evaluation  process  considering  and  calculating  an  overall  performance 
value  or  score  may  let  the  decision  makers  gain  more  insights  into  the  system  they 
govern.  Thus,  overall  perfonnance  metrics  may  play  a  more  effective  role  in  the  decision¬ 
making  process  than  the  individual  metrics. 

Limitations 

First  of  all,  PBL  is  a  new  and  sensitive  approach,  because  it  both  deals  with 
critical  weapon  systems  and  there  have  been  some  failures  experienced  so  far.  Thus, 
finding  a  sponsorship  and  gathering  specific  raw  data  about  PBL  systems  were  limited 
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throughout  the  current  study.  Although  a  theoretical  model  is  built  in  the  current  study, 
only  published  sources  of  data  were  obtrained. 

Secondly,  the  best  practices  are  selected  for  the  publications  generally  and  they 
included  very  limited  and  aggregated  data  such  as  annual  averages  of  related  metrics,  or 
comparisons  of  only  some  of  the  metrics  that  were  defined  in  the  contracts. 

Future  Studies 

First  of  all,  the  4SPEM  model  should  be  tested  via  more  real  life  data.  An 
application  of  this  model  to  a  PBL  program  that  has  intense  real  life  data  via  a 
sponsorship  may  prove  useful. 

Secondly,  the  best  practices  of  PBL  may  be  useful  to  concentrate  on  the  efforts, 
however;  considering  other  practices  than  the  best  ones  and  application  of  this  model  on 
them  may  help  acquisition  professionals  to  gain  more  insight  into  PBL  contracts. 

Conclusion 

PBL  is  an  innovative  approach  to  satisfy  the  highly  elastic  needs  of  the  warfighter 
and  a  new  approach,  which  may  be  adopted  as  an  evaluation  process  in  PBL  is  proposed 
in  this  thesis.  The  4SPEM  attempts  to  fill  some  of  the  gaps  in  the  area  of  evaluation 
processes  and  may  be  a  promising  model.  However,  further  studies  depending  on  intense 
real  life  raw  data  via  the  sponsorships  of  the  PBL  Program  Offices  are  needed  to 
investigate  the  limitations  and  true  benefits  of  this  model. 
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APPENDIX  A 


Calculation  of  CQTFs  and  OPMs  for  Case  1 

) 

Calculation  of  CQTFs  and  OPMs  for  Contractual  Values 
(The  calculations  are  depicted  for  4  terms 
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Calculation  of  CQTFs  and  OPMs  for  Assessed  Values 
(The  calculations  are  depicted  for  4  terms) 
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APPENDIX  B 


Calculation  of  DLTF’s  and  OPMs  for  Case  1 

Calculation  of  DLTF’s  and  OPMs  for  Contractual  Values 
(The  calculations  are  depicted  for  4  terms) 
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Calculation  ofDLTF’s  and  OPMs  for  Assessed  Values 
(The  calculations  are  depicted  for  4  terms 


A1  := 


0  1 
1  1 


C101  := 


92.1 

89.5 


C102  := 


1.8 


Al_  1  = 


-1  1 

U  o 


A1  !-C101  = 


-2.6 

(92.1 


A1  !-C102  = 


0.2 

1.8 


SMA101(t)  :=  -2.6t  +  92.1  AWCTlOl(t)  :=  0.2t  +  1.8 


r  1  n 

SMA101(t)  d 

° 


1 


AWCTlOl(t) 


ti=  47.834 


2nd  Term 


Since  DLTFs  are  considered,  A1=A2=A3=A4 


A1  := 


0  0 
1  b 


C103  := 


89.5 

91.1 


C104  := 


Al_1  = 


f-1  ‘1 

-  1  f  1.6  ^ 

A1  -C103  = 

-  1  ( -0.9^ 

A1  -C104  = 

1  Oj 

\89-57 

l  2  J 

SMA102(t)  :=  1.6t  +  89.5  AWCT102(t)  :=  -0.9t  +  2 


r  1 


r  1 


SMA102(t)  d 


1 


AWCT102(t) 


ti=  59.983 


101 


3rd  Term 


A1  := 


0  1 
1  1 


C105  := 


91.1 

86.4 


C106  := 


1.1 

2.3 


Al"1  = 


-1  1 

1  0 


-  l  I  -4.7 

Al  -C105  = 

91.1 


Al  !-C106 


SMA103(t)  :=  -4.7t  +  91.1 


AWCT103(t)  :=  1.2t  +  1.1 


SMA103(t)  d 


1 


AWCT103(t) 


(1=  54.552 


4th  Tenn 


Al  := 


0  1 
1  1 


C107  := 


86.4 

100 


C108  := 


2.3 

1.7 


-1  l-l  1 
Al  = 

1  0 


_  i  i  13.6 

Al  -C107  = 

86.4 


Al  !-C108  = 


SMA104(t)  :=  13. 6t  +  86.4 


AWCT104(t)  :=  -0.6t  +  2.3 


r  i 

SMA104(t) 


AWCT104(t) 

J0 


(1=46.954 


102 


APPENDIX  C 


Sensitivity  Analysis  for  the  Conflict  Situation 

4th  Term 
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APPENDIX  D 


Calculation  of  DLTF’s  and  OPMs  for  Case  2 


Calculation  of  DLTF’s  and  OPMs  for  Contractual  Values 
(The  calculations  are  depicted  for  4  terms 
1st  Tenn 


A101 


(A101) 


f-l 

vl 


C101  := 


- 1 


69  ) 

C102  := 

71.7 J 

Os) 

(2.1\ 

1-C102  = 

•C101  = 

(A101)” 

GSAAClOl(t)  :=  2.7t  +  69 
IEC101(t)  :=  8t  +  67 
FHAClOl(t)  :=  95 
MICAPEClOl(t)  :=  80 
ADMSEClOl(t)  :=  98 


- 1 

- 1 

•  1 

- 1 

r1 

GSAAClOl(t)  a 

FHAClOl(t)  d 

MICAPEClOl(t)  d 

ADMSEClOl(t)  ti 

IEC101(t)  d=  3.72  x  109 

J  0  J 

0  J 

0  J 

0  J 

0 

2nd  Term 


A 102 


(A  102) 


f-l 

vl 


C103  := 


-  1 


71.7  | 

C104  := 

74.2  J 

UJ 

(  2.5  ^ 

'■C104  = 

•C103  = 

(A  102)“ 

,  71.7, 


GSAAC102(t)  :=  2.5t  +  71.7 
IEC102(t)  :=  3t  +  75 
FHAC102(t)  :=  95 
MICAPEC102(t)  :=  80 
ADMSEC102(t)  :=  98 


-1 

r1 

r1 

f1  1 

r1 

GSAAC102(t)  a 

FHAC102(t)  a 

MICAPEC102(t)  a 

ADMSEC102(t)  a 

IEC102(t)  a  =  4.156  X  lo9 

0  0  0  0 


105 


Calculation  ofDLTF’s  and  OPMs  for  Assessed  Values 
(The  calculations  are  depicted  for  4  terms) 
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APPENDIX  E 


The  Blue  Dart 

1st  Lt  Ferit  BUYUKGURAL,  Student,  AFIT 
Word  count:  823 

Beyond  any  question  the  most  remarkable  global  developments  of  the  end  of  the 
20th  Century  and  the  beginning  of  the  2 1st  Century  were:  the  ending  of  the  Cold  War  and 
the  9/11  attacks.  The  Cold  War  resulted  in  the  dissolution  of  the  Union  of  Soviet 
Socialist  Republics  (USSR)  and  the  Warsaw  Treaty.  The  collapse  of  the  USSR  resulted  in 
large  part  due  to  the  misallocation  of  their  resources  and  inefficiencies  of  their  system. 
Although  there  was  one  enemy,  it  was  spread  out  geographically,  making  it  hard  for  the 
USSR  to  carry  the  heavy  burden  of  logistics  support  for  its  system.  History  clarifies  that 
using  a  large  conventional  army  makes  war  very  expensive  and  requires  a  large  support 
system  to  maintain  effectiveness. 

The  terrorist  attacks  of  September  11,  2001  (9/1 1)  resulted  in  implementation  of 
the  Global  War  on  Terrorism  (GWOT)  by  the  US  and  its  allies.  The  concept  of  GWOT 
represents  shifting  from  traditional  anny  warfare  to  asymmetric  warfare.  Briefly,  there’s 
no  enemy  country  but  there’s  an  enemy,  there  are  no  borders  of  conflict  zone,  but 
everywhere  is  potentially  a  conflict  zone  and  showing  presence  of  big  conventional 
armed  forces  in  an  area  is  no  more  a  show  of  superiority,  but  becoming  a  bigger  target  for 
the  shadow  enemy.  The  necessities  of  the  new  era  mandated  every  country  to  allocate 
their  resources  relative  to  the  new  war  concepts  that  are  evolved  from  the  previous  ones 
and  Performance-Based  Logistics  (PBL)  is  introduced  under  these  conditions. 
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PBL  is  buying  performance  instead  of  buying  materials  and  it  is  DoD's 
preferred  product  support  approach  in  order  to  satisfy  vitally  important  war-fighter  needs. 
The  mindset  behind  the  PBL  approach  is:  if  the  contractor  knows  how  to  produce  a  high- 
technology  product  or  service,  then  s/he  knows  how  to  sustain  it  better  than  the  customer. 
But  to  what  extend  can  we  trust  contractors  and  authorize  them  for  our  weapon  system? 
Additionally,  the  elasticity  of  the  needs  makes  it  impossible  to  build  a  one-fits-all  PBL 
contract  template  that  would  be  applicable  for  a  number  of  scenarios.  If  there’s  no 
template,  how  should  it  be  implemented  correctly?  How  can  anybody  be  sure  about 
obtaining  the  best  results  via  using  PBL? 

The  needs  should  be  defined  clearly  by  the  operational  units  and  the  supporting 
units.  Then  the  system  should  be  monitored  and  governed  accordingly.  Basically,  we 
should  define  the  metrics  concerned  with  the  desired  outcomes,  come  to  an  agreement 
with  the  contractor/s  about  the  metrics  that  we  define,  collect  the  data  of  that  metrics,  and 
evaluate  them  in  order  to  monitor  our  system.  Otherwise,  we  would  not  be  able  to 
manage  our  system  in  a  healthy  manner. 

Both  economic  metrics  such  as:  Cost  per  Unit  Usage  and  non-economic  metrics 
such  as:  Availability,  Reliability,  Logistics  Foot  Print  and  Logistics  Response  Time  can 
be  defined  and  tailored  according  to  each  individual  program.  However,  evaluation  of  the 
success  of  a  PBL  contract  gets  harder  proportional  to  the  increased  number  of  different 
types  of  metrics.  According  to  the  experts’  opinions,  in  most  of  the  PBL  contracts,  the 
metrics  are  evaluated  by  collecting  assessed  values  of  the  metrics  within  the 
contractually-defined  terms,  (e.g.,  average  monthly  values)  and  comparing  assessed 
values  of  the  metrics  with  the  contractually  targeted  numbers  of  that  metrics  individually. 
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However,  there  are  some  issues  concerned  with  this  approach.  First,  every  pair  of 
comparisons  includes  a  risk  of  making  misjudgments  statistically,  and  this  risk  increases 
as  the  number  of  pairs  increase.  Second,  there  may  be  conflicts  between  the  metrics  such 
as  some  metrics  may  be  over  performing  while  the  others  are  underperforming,  and  it 
complicates  the  situation  for  the  decision  makers.  Most  recently,  the  US  Government 
Accountability  Office  (US  GAO)  declared  that  although  many  perfonnance  metrics  have 
been  used  in  contracts,  they  have  not  been  used  in  the  decision  making  processes. 

This  thesis  investigates  and  proposes  an  alternative  approach  to  monitor  the 
Perfonnance  Based  Logistics  (PBL)  supported  systems  both  at  the  individual  metrics 
level  and  at  the  whole  system  level.  The  proposed  approach  is  the  4-Step  Process 
Evaluation  Model  (4-SPEM),  consisting  of  1)  Building  the  Current  Evaluation  Model 
(CEM),  2)  Building  the  Mathematical  Model  (MM),  3)  Building  the  Integrated  Model 
(IM)  and  4)Analysis  and  Interpretation.  Briefly,  the  defined  metrics  of  the  PBL  contracts 
are  assessed  against  individual  target  performance  levels  by  time  period  (CEM).  The 
individual  metrics  are  then  integrated  into  a  mathematical  model  (MM).  The  CEM  and 
MM  are  integrated  to  assess  results  and  gain  insights  at  both  the  individual  metric  and 
overall  system  perfonnance  levels.  Analysis  could  include  forecasting  future 
performance  based  on  perfonnance  to  targets  to  identify  potential  issues. 

This  approach  may  be  used  in  operational-level  decisions  such  as  reward  and 
penalty  decisions  within  a  contract  or  strategic-level  decisions  such  as  extending  the 
contract  with  the  same  contractor,  tenninating  the  contract  and  negotiating 
with  other  contractors.  In  a  more  general  sense,  this  approach  could  be  applied  to  other 
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management  and  monitoring  decisions  that  involve  multiple  metrics  with  different 
measurement  characteristics  across  multiple  time  periods. 
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